Oct 31 01:17:26.251282 ip-10-0-133-98 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 31 01:17:26.251295 ip-10-0-133-98 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 31 01:17:26.251305 ip-10-0-133-98 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 31 01:17:26.251646 ip-10-0-133-98 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 31 01:17:36.441462 ip-10-0-133-98 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 31 01:17:36.441479 ip-10-0-133-98 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 5f1a19c8732448a28b26c1e71927a23d --
Oct 31 01:19:32.011798 ip-10-0-133-98 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 01:19:32.469782 ip-10-0-133-98 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:19:32.469782 ip-10-0-133-98 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 01:19:32.469782 ip-10-0-133-98 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:19:32.469782 ip-10-0-133-98 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:19:32.473804 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473714    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:19:32.473804 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473803    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:19:32.473804 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473807    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:19:32.473904 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473811    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:19:32.473904 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473814    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:19:32.473904 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473817    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:19:32.473904 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473819    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:19:32.473904 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473822    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:19:32.473904 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473825    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:19:32.473904 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473828    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:19:32.473904 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473831    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:19:32.473904 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473833    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:19:32.473904 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473836    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:19:32.473904 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473839    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:19:32.473904 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473841    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:19:32.473904 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473845    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:19:32.473904 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473847    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:19:32.473904 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473851    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:19:32.473904 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473853    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:19:32.473904 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473856    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:19:32.473904 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473859    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:19:32.473904 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473861    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:19:32.474341 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473864    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:19:32.474341 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473867    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:19:32.474341 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473874    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:19:32.474341 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473877    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:19:32.474341 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473879    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:19:32.474341 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473882    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:19:32.474341 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473885    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:19:32.474341 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473888    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:19:32.474341 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473890    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:19:32.474341 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473893    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:19:32.474341 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473896    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:19:32.474341 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473899    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:19:32.474341 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473902    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:19:32.474341 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473904    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:19:32.474341 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473909    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:19:32.474341 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473912    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:19:32.474341 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473915    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:19:32.474341 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473918    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:19:32.474341 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473920    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:19:32.474341 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473923    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:19:32.474856 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473926    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:19:32.474856 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473928    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:19:32.474856 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473931    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:19:32.474856 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473934    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:19:32.474856 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473937    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:19:32.474856 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473939    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:19:32.474856 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473942    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:19:32.474856 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473945    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:19:32.474856 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473948    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:19:32.474856 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473950    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:19:32.474856 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473953    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:19:32.474856 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473955    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:19:32.474856 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473958    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:19:32.474856 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473961    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:19:32.474856 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473964    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:19:32.474856 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473967    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:19:32.474856 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473970    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:19:32.474856 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473975    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:19:32.474856 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473978    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:19:32.475303 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473981    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:19:32.475303 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473983    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:19:32.475303 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473986    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:19:32.475303 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473990    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:19:32.475303 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473995    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:19:32.475303 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.473998    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:19:32.475303 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474001    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:19:32.475303 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474004    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:19:32.475303 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474007    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:19:32.475303 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474010    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:19:32.475303 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474013    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:19:32.475303 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474016    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:19:32.475303 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474019    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:19:32.475303 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474021    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:19:32.475303 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474024    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:19:32.475303 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474026    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:19:32.475303 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474029    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:19:32.475303 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474032    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:19:32.475303 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474035    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:19:32.475785 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474037    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:19:32.475785 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474040    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:19:32.475785 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474042    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:19:32.475785 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474045    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:19:32.475785 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474049    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:19:32.475785 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474052    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:19:32.475785 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474055    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:19:32.475785 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474058    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:19:32.475785 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474061    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:19:32.475785 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474068    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:19:32.475785 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474071    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:19:32.475785 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474073    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:19:32.475785 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474076    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:19:32.475785 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474079    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:19:32.475785 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474081    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:19:32.475785 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474084    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:19:32.475785 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474087    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:19:32.475785 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474089    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:19:32.475785 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474093    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:19:32.476225 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474098    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:19:32.476225 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474101    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:19:32.476225 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474105    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:19:32.476225 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474108    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:19:32.476225 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474110    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:19:32.476225 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474115    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:19:32.476225 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474117    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:19:32.476225 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474932    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:19:32.476225 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474939    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:19:32.476225 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474942    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:19:32.476225 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474945    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:19:32.476225 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474947    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:19:32.476225 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474950    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:19:32.476225 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474954    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:19:32.476225 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474957    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:19:32.476225 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474960    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:19:32.476225 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474963    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:19:32.476225 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474966    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:19:32.476225 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474970    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:19:32.476225 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474981    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:19:32.476722 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474984    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:19:32.476722 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474987    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:19:32.476722 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474990    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:19:32.476722 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474993    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:19:32.476722 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474996    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:19:32.476722 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.474999    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:19:32.476722 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475002    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:19:32.476722 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475005    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:19:32.476722 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475007    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:19:32.476722 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475010    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:19:32.476722 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475013    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:19:32.476722 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475015    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:19:32.476722 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475018    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:19:32.476722 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475021    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:19:32.476722 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475024    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:19:32.476722 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475027    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:19:32.476722 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475029    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:19:32.476722 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475032    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:19:32.476722 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475035    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:19:32.476722 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475037    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:19:32.477195 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475040    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:19:32.477195 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475043    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:19:32.477195 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475046    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:19:32.477195 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475048    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:19:32.477195 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475051    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:19:32.477195 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475053    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:19:32.477195 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475056    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:19:32.477195 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475059    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:19:32.477195 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475063    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:19:32.477195 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475066    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:19:32.477195 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475069    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:19:32.477195 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475072    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:19:32.477195 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475075    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:19:32.477195 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475078    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:19:32.477195 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475081    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:19:32.477195 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475083    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:19:32.477195 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475086    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:19:32.477195 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475089    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:19:32.477195 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475094    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:19:32.477661 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475097    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:19:32.477661 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475100    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:19:32.477661 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475104    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:19:32.477661 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475107    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:19:32.477661 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475109    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:19:32.477661 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475112    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:19:32.477661 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475115    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:19:32.477661 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475118    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:19:32.477661 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475120    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:19:32.477661 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475123    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:19:32.477661 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475126    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:19:32.477661 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475128    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:19:32.477661 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475131    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:19:32.477661 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475133    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:19:32.477661 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475136    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:19:32.477661 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475139    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:19:32.477661 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475141    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:19:32.477661 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475144    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:19:32.477661 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475146    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:19:32.478115 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475149    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:19:32.478115 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475152    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:19:32.478115 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475154    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:19:32.478115 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475159    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:19:32.478115 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475162    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:19:32.478115 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475165    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:19:32.478115 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475167    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:19:32.478115 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475170    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:19:32.478115 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475173    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:19:32.478115 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475175    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:19:32.478115 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475178    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:19:32.478115 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475182    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:19:32.478115 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475184    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:19:32.478115 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475187    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:19:32.478115 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475190    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:19:32.478115 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475192    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:19:32.478115 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475195    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:19:32.478115 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475198    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:19:32.478115 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475200    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:19:32.478605 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475203    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:19:32.478605 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475206    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:19:32.478605 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475208    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:19:32.478605 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475211    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:19:32.478605 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475213    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:19:32.478605 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475216    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:19:32.478605 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475219    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:19:32.478605 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475221    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:19:32.478605 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475224    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:19:32.478605 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475228    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:19:32.478605 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475230    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:19:32.478605 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475233    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:19:32.478605 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475235    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:19:32.478605 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475238    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:19:32.478605 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475240    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:19:32.478605 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475243    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:19:32.478605 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475748    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:19:32.478605 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475754    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:19:32.478605 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475757    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:19:32.479061 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475760    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:19:32.479061 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475763    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:19:32.479061 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475766    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:19:32.479061 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475769    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:19:32.479061 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475771    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:19:32.479061 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475774    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:19:32.479061 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475778    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:19:32.479061 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475781    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:19:32.479061 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475783    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:19:32.479061 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475787    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:19:32.479061 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475790    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:19:32.479061 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475793    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:19:32.479061 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475796    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:19:32.479061 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475799    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:19:32.479061 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475803    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:19:32.479061 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475807    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:19:32.479061 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475810    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:19:32.479061 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475813    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:19:32.479061 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475816    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:19:32.479529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475818    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:19:32.479529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475821    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:19:32.479529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475824    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:19:32.479529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475827    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:19:32.479529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475829    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:19:32.479529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475832    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:19:32.479529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475835    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:19:32.479529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475838    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:19:32.479529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475840    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:19:32.479529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475843    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:19:32.479529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475846    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:19:32.479529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475849    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:19:32.479529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475852    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:19:32.479529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475855    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:19:32.479529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475857    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:19:32.479529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475860    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:19:32.479529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475863    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:19:32.479529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475866    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:19:32.479529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475868    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:19:32.479529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475871    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:19:32.480036 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475874    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:19:32.480036 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475876    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:19:32.480036 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475881    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:19:32.480036 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475884    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:19:32.480036 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475886    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:19:32.480036 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475889    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:19:32.480036 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475891    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:19:32.480036 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475894    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:19:32.480036 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475896    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:19:32.480036 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475899    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:19:32.480036 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475902    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:19:32.480036 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475904    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:19:32.480036 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475907    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:19:32.480036 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475910    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:19:32.480036 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475913    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:19:32.480036 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475915    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:19:32.480036 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475918    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:19:32.480036 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475921    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:19:32.480036 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475923    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:19:32.480583 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475926    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:19:32.480583 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475928    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:19:32.480583 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475931    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:19:32.480583 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475934    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:19:32.480583 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475936    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:19:32.480583 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475939    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:19:32.480583 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475942    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:19:32.480583 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475944    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:19:32.480583 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475948    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:19:32.480583 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475952    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:19:32.480583 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475955    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:19:32.480583 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475958    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:19:32.480583 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475961    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:19:32.480583 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475964    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:19:32.480583 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475966    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:19:32.480583 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475970    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:19:32.480583 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475973    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:19:32.480583 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475975    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:19:32.480583 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475978    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:19:32.481032 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475980    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:19:32.481032 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475983    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:19:32.481032 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475986    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:19:32.481032 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475988    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:19:32.481032 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475991    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:19:32.481032 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475993    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:19:32.481032 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475996    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:19:32.481032 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.475998    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:19:32.481032 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.476001    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:19:32.481032 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.476003    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:19:32.481032 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.476006    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:19:32.481032 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.476009    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:19:32.481032 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.476012    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:19:32.481032 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.476014    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:19:32.481032 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.476017    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:19:32.481032 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.476019    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:19:32.481032 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.476022    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:19:32.481032 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.476025    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:19:32.481032 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.476028    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:19:32.481032 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.476030    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:19:32.481576 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.476033    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:19:32.481576 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.476035    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:19:32.481576 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.476038    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:19:32.481576 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.476041    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:19:32.481576 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.476044    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:19:32.481576 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.476047    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:19:32.481576 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.476963    2578 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 01:19:32.481576 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.476975    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 01:19:32.481576 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.476984    2578 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 01:19:32.481576 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.476989    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 01:19:32.481576 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.476994    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 01:19:32.481576 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.476997    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 01:19:32.481576 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477002    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 01:19:32.481576 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477006    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 01:19:32.481576 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477010    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 01:19:32.481576 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477013    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 01:19:32.481576 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477016    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 01:19:32.481576 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477020    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 01:19:32.481576 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477024    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 01:19:32.481576 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477028    2578 flags.go:64] FLAG: --cgroup-root=""
Oct 31 01:19:32.481576 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477031    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 01:19:32.481576 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477034    2578 flags.go:64] FLAG: --client-ca-file=""
Oct 31 01:19:32.481576 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477037    2578 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 01:19:32.482536 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477040    2578 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 01:19:32.482536 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477046    2578 flags.go:64] FLAG: --cluster-domain=""
Oct 31 01:19:32.482536 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477049    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 01:19:32.482536 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477052    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 01:19:32.482536 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477056    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 01:19:32.482536 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477059    2578 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 01:19:32.482536 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477063    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 01:19:32.482536 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477067    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 01:19:32.482536 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477070    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 01:19:32.482536 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477073    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 01:19:32.482536 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477077    2578 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 01:19:32.482536 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477080    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 01:19:32.482536 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477083    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 01:19:32.482536 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477087    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 01:19:32.482536 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477090    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 01:19:32.482536 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477094    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 01:19:32.482536 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477098    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 01:19:32.482536 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477102    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 01:19:32.482536 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477106    2578 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 01:19:32.482536 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477109    2578 flags.go:64] FLAG: --enable-server="true"
Oct 31 01:19:32.482536 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477112    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 01:19:32.482536 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477117    2578 flags.go:64] FLAG: --event-burst="100"
Oct 31 01:19:32.482536 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477120    2578 flags.go:64] FLAG: --event-qps="50"
Oct 31 01:19:32.482536 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477123    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 01:19:32.483161 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477126    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 01:19:32.483161 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477129    2578 flags.go:64] FLAG: --eviction-hard=""
Oct 31 01:19:32.483161 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477134    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 01:19:32.483161 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477137    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 01:19:32.483161 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477141    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 01:19:32.483161 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477144    2578 flags.go:64] FLAG: --eviction-soft=""
Oct 31 01:19:32.483161 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477147    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 01:19:32.483161 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477150    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 01:19:32.483161 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477153    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 01:19:32.483161 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477156    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 01:19:32.483161 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477159    2578 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 01:19:32.483161 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477162    2578 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 01:19:32.483161 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477167    2578 flags.go:64] FLAG: --feature-gates=""
Oct 31 01:19:32.483161 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477171    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 01:19:32.483161 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477175    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 01:19:32.483161 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477178    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 01:19:32.483161 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477181    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 01:19:32.483161 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477185    2578 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 01:19:32.483161 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477188    2578 flags.go:64] FLAG: --help="false"
Oct 31 01:19:32.483161 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477191    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:19:32.483161 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477195    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 01:19:32.483161 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477198    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 01:19:32.483161 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477201    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 01:19:32.483734 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477205    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 01:19:32.483734 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477209    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 01:19:32.483734 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477213    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 01:19:32.483734 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477217    2578 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 01:19:32.483734 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477220    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 01:19:32.483734 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477223    2578 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 01:19:32.483734 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477226    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 01:19:32.483734 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477230    2578 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 01:19:32.483734 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477233    2578 flags.go:64] FLAG: --kube-reserved=""
Oct 31 01:19:32.483734 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477236    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 01:19:32.483734 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477239    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 01:19:32.483734 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477243    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 01:19:32.483734 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477246    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 01:19:32.483734 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477248    2578 flags.go:64] FLAG: --lock-file=""
Oct 31 01:19:32.483734 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477251    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 01:19:32.483734 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477255    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 01:19:32.483734 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477258    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 01:19:32.483734 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477264    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 01:19:32.483734 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477267    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 01:19:32.483734 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477270    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 01:19:32.483734 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477273    2578 flags.go:64] FLAG: --logging-format="text"
Oct 31 01:19:32.483734 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477278    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 01:19:32.483734 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477281    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 01:19:32.483734 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477284    2578 flags.go:64] FLAG: --manifest-url=""
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477287    2578 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477292    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477296    2578 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477300    2578 flags.go:64] FLAG: --max-pods="110"
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477303    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477306    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477309    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477313    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477316    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477319    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477322    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477329    2578 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477333    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477336    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477339    2578 flags.go:64] FLAG: --pod-cidr=""
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477342    2578 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477345    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477349    2578 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477352    2578 flags.go:64] FLAG: --port="10250"
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477355    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477358    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-000fec9b7531646b6"
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477362    2578 flags.go:64] FLAG: --qos-reserved=""
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477365    2578 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477368    2578 flags.go:64] FLAG: --register-node="true"
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477371    2578 flags.go:64] FLAG: --register-with-taints=""
Oct 31 01:19:32.484330 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477375    2578 flags.go:64] FLAG: --registry-burst="10"
Oct 31 01:19:32.484967 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477378    2578 flags.go:64] FLAG: --registry-qps="5"
Oct 31 01:19:32.484967 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477381    2578 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 01:19:32.484967 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477384    2578 flags.go:64] FLAG: --reserved-memory=""
Oct 31 01:19:32.484967 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477390    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 01:19:32.484967 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477393    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 01:19:32.484967 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477396    2578 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 01:19:32.484967 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477399    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 01:19:32.484967 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477402    2578 flags.go:64] FLAG: --runonce="false"
Oct 31 01:19:32.484967 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477405    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 01:19:32.484967 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477408    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 01:19:32.484967 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477412    2578 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 01:19:32.484967 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477415    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 01:19:32.484967 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477417    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 01:19:32.484967 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477421    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 01:19:32.484967 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477424    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 01:19:32.484967 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477428    2578 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 01:19:32.484967 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477431    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 01:19:32.484967 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477434    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 01:19:32.484967 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477437    2578 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 01:19:32.484967 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477441    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 01:19:32.484967 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477453    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 01:19:32.484967 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477457    2578 flags.go:64] FLAG: --system-cgroups=""
Oct 31 01:19:32.484967 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477460    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 01:19:32.484967 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477465    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 01:19:32.484967 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477468    2578 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 01:19:32.485624 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477471    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 01:19:32.485624 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477476    2578 flags.go:64] FLAG: --tls-min-version=""
Oct 31 01:19:32.485624 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477479    2578 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 01:19:32.485624 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477482    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 01:19:32.485624 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477485    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 01:19:32.485624 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477488    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 01:19:32.485624 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477491    2578 flags.go:64] FLAG: --v="2"
Oct 31 01:19:32.485624 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477496    2578 flags.go:64] FLAG: --version="false"
Oct 31 01:19:32.485624 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477500    2578 flags.go:64] FLAG: --vmodule=""
Oct 31 01:19:32.485624 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477504    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 01:19:32.485624 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.477510    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 01:19:32.485624 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477648    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:19:32.485624 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477651    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:19:32.485624 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477654    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:19:32.485624 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477657    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:19:32.485624 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477660    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:19:32.485624 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477663    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:19:32.485624 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477666    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:19:32.485624 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477669    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:19:32.485624 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477672    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:19:32.485624 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477675    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:19:32.485624 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477678    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:19:32.486167 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477681    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:19:32.486167 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477684    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:19:32.486167 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477687    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:19:32.486167 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477689    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:19:32.486167 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477692    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:19:32.486167 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477695    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:19:32.486167 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477698    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:19:32.486167 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477700    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:19:32.486167 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477704    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:19:32.486167 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477707    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:19:32.486167 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477709    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:19:32.486167 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477712    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:19:32.486167 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477715    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:19:32.486167 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477717    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:19:32.486167 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477720    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:19:32.486167 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477723    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:19:32.486167 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477725    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:19:32.486167 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477729    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:19:32.486167 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477734    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:19:32.486911 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477737    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:19:32.486911 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477741    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:19:32.486911 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477743    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:19:32.486911 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477746    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:19:32.486911 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477749    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:19:32.486911 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477752    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:19:32.486911 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477754    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:19:32.486911 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477757    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:19:32.486911 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477760    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:19:32.486911 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477763    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:19:32.486911 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477766    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:19:32.486911 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477768    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:19:32.486911 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477772    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:19:32.486911 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477775    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:19:32.486911 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477777    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:19:32.486911 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477782    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:19:32.486911 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477786    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:19:32.486911 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477789    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:19:32.486911 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477792    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:19:32.486911 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477795    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:19:32.487581 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477798    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:19:32.487581 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477801    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:19:32.487581 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477803    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:19:32.487581 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477806    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:19:32.487581 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477809    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:19:32.487581 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477812    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:19:32.487581 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477814    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:19:32.487581 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477817    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:19:32.487581 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477820    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:19:32.487581 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477822    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:19:32.487581 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477825    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:19:32.487581 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477828    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:19:32.487581 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477830    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:19:32.487581 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477835    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:19:32.487581 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477837    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:19:32.487581 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477840    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:19:32.487581 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477843    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:19:32.487581 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477846    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:19:32.487581 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477848    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:19:32.488038 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477851    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:19:32.488038 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477854    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:19:32.488038 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477856    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:19:32.488038 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477859    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:19:32.488038 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477862    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:19:32.488038 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477866    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:19:32.488038 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477869    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:19:32.488038 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477872    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:19:32.488038 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477875    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:19:32.488038 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477877    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:19:32.488038 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477880    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:19:32.488038 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477883    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:19:32.488038 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477885    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:19:32.488038 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477889    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:19:32.488038 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477892    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:19:32.488038 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477894    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:19:32.488038 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477897    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:19:32.488038 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477900    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:19:32.488038 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477903    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:19:32.488038 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477905    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:19:32.488529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477908    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:19:32.488529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477911    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:19:32.488529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477913    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:19:32.488529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477916    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:19:32.488529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477918    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:19:32.488529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477921    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:19:32.488529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477925    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:19:32.488529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477928    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:19:32.488529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477930    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:19:32.488529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477933    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:19:32.488529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477936    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:19:32.488529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477939    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:19:32.488529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477941    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:19:32.488529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477944    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:19:32.488529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477946    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:19:32.488529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477967    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:19:32.488529 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.477972    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:19:32.488937 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.478000    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:19:32.488937 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478037    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:19:32.488937 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478041    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:19:32.488937 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478044    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:19:32.488937 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478047    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:19:32.488937 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478050    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:19:32.488937 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478053    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:19:32.488937 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478056    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:19:32.488937 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478058    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:19:32.488937 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478062    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:19:32.488937 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478065    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:19:32.488937 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478068    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:19:32.488937 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478070    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:19:32.488937 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478073    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:19:32.488937 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478075    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:19:32.488937 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478078    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:19:32.489312 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478081    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:19:32.489312 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478084    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:19:32.489312 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478086    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:19:32.489312 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478089    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:19:32.489312 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478091    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:19:32.489312 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478094    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:19:32.489312 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478099    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:19:32.489312 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478101    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:19:32.489312 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478104    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:19:32.489312 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478106    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:19:32.489312 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478109    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:19:32.489312 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478112    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:19:32.489312 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478115    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:19:32.489312 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478117    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:19:32.489312 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478121    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:19:32.489312 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478125    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:19:32.489312 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478129    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:19:32.489312 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478132    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:19:32.489312 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478135    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:19:32.489774 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478137    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:19:32.489774 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478140    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:19:32.489774 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478143    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:19:32.489774 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478146    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:19:32.489774 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478148    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:19:32.489774 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478151    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:19:32.489774 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478154    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:19:32.489774 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478157    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:19:32.489774 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478160    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:19:32.489774 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478163    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:19:32.489774 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478165    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:19:32.489774 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478168    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:19:32.489774 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478172    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:19:32.489774 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478176    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:19:32.489774 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478180    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:19:32.489774 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478183    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:19:32.489774 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478185    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:19:32.489774 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478188    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:19:32.489774 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478191    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:19:32.489774 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478194    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:19:32.490240 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478196    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:19:32.490240 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478199    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:19:32.490240 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478202    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:19:32.490240 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478204    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:19:32.490240 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478207    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:19:32.490240 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478209    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:19:32.490240 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478212    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:19:32.490240 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478215    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:19:32.490240 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478217    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:19:32.490240 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478221    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:19:32.490240 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478224    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:19:32.490240 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478226    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:19:32.490240 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478229    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:19:32.490240 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478231    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:19:32.490240 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478234    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:19:32.490240 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478237    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:19:32.490240 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478239    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:19:32.490240 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478242    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:19:32.490240 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478245    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:19:32.490753 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478247    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:19:32.490753 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478250    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:19:32.490753 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478253    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:19:32.490753 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478256    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:19:32.490753 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478258    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:19:32.490753 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478261    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:19:32.490753 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478264    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:19:32.490753 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478266    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:19:32.490753 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478269    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:19:32.490753 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478271    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:19:32.490753 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478274    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:19:32.490753 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478277    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:19:32.490753 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478279    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:19:32.490753 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478282    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:19:32.490753 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478284    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:19:32.490753 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478287    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:19:32.490753 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478289    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:19:32.490753 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478292    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:19:32.490753 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478294    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:19:32.490753 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478297    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:19:32.491262 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478299    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:19:32.491262 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478302    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:19:32.491262 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478306    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:19:32.491262 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478308    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:19:32.491262 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478311    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:19:32.491262 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478314    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:19:32.491262 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478317    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:19:32.491262 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478319    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:19:32.491262 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478322    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:19:32.491262 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478324    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:19:32.491262 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478327    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:19:32.491262 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478330    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:19:32.491262 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.478332    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:19:32.491262 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.488281    2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 01:19:32.491262 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.488299    2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 01:19:32.491262 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.489036    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 01:19:32.491262 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.489044    2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 01:19:32.491262 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489110    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:19:32.491262 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489117    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:19:32.491727 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489121    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:19:32.491727 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489124    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:19:32.491727 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489128    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:19:32.491727 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489130    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:19:32.491727 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489134    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:19:32.491727 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489136    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:19:32.491727 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489139    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:19:32.491727 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489142    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:19:32.491727 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489145    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:19:32.491727 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489148    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:19:32.491727 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489151    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:19:32.491727 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489153    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:19:32.491727 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489156    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:19:32.491727 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489158    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:19:32.491727 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489164    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:19:32.491727 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489169    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:19:32.491727 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489173    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:19:32.491727 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489176    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:19:32.491727 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489179    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:19:32.492166 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489181    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:19:32.492166 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489184    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:19:32.492166 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489187    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:19:32.492166 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489190    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:19:32.492166 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489192    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:19:32.492166 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489195    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:19:32.492166 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489197    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:19:32.492166 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489200    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:19:32.492166 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489204    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:19:32.492166 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489207    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:19:32.492166 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489210    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:19:32.492166 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489213    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:19:32.492166 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489215    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:19:32.492166 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489218    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:19:32.492166 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489221    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:19:32.492166 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489224    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:19:32.492166 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489226    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:19:32.492166 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489229    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:19:32.492166 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489232    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:19:32.492166 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489234    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:19:32.492820 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489237    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:19:32.492820 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489240    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:19:32.492820 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489242    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:19:32.492820 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489245    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:19:32.492820 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489248    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:19:32.492820 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489250    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:19:32.492820 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489253    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:19:32.492820 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489256    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:19:32.492820 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489259    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:19:32.492820 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489262    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:19:32.492820 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489265    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:19:32.492820 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489267    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:19:32.492820 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489270    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:19:32.492820 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489273    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:19:32.492820 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489275    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:19:32.492820 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489278    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:19:32.492820 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489280    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:19:32.492820 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489283    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:19:32.492820 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489286    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:19:32.492820 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489288    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:19:32.493290 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489292    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:19:32.493290 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489295    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:19:32.493290 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489298    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:19:32.493290 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489301    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:19:32.493290 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489304    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:19:32.493290 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489306    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:19:32.493290 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489309    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:19:32.493290 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489311    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:19:32.493290 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489314    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:19:32.493290 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489317    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:19:32.493290 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489320    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:19:32.493290 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489323    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:19:32.493290 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489326    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:19:32.493290 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489329    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:19:32.493290 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489332    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:19:32.493290 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489334    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:19:32.493290 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489337    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:19:32.493290 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489340    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:19:32.493290 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489343    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:19:32.493761 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489346    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:19:32.493761 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489349    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:19:32.493761 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489351    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:19:32.493761 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489354    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:19:32.493761 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489358    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:19:32.493761 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489362    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:19:32.493761 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489366    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:19:32.493761 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489369    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:19:32.493761 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489371    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:19:32.493761 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489374    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:19:32.493761 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489376    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:19:32.493761 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489379    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:19:32.493761 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489381    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:19:32.493761 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489386    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:19:32.493761 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489388    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:19:32.493761 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489391    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:19:32.493761 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489394    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:19:32.493761 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489397    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:19:32.493761 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489400    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:19:32.494203 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489402    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:19:32.494203 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489405    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:19:32.494203 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489408    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:19:32.494203 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489410    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:19:32.494203 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489413    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:19:32.494203 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489416    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:19:32.494203 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489418    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:19:32.494203 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.489463    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:19:32.494203 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489622    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:19:32.494203 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489629    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:19:32.494203 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489632    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:19:32.494203 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489635    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:19:32.494203 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489638    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:19:32.494203 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489641    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:19:32.494203 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489644    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:19:32.494203 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489647    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:19:32.494602 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489650    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:19:32.494602 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489652    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:19:32.494602 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489655    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:19:32.494602 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489658    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:19:32.494602 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489660    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:19:32.494602 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489663    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:19:32.494602 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489666    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:19:32.494602 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489669    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:19:32.494602 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489671    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:19:32.494602 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489674    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:19:32.494602 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489677    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:19:32.494602 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489680    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:19:32.494602 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489683    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:19:32.494602 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489686    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:19:32.494602 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489688    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:19:32.494602 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489691    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:19:32.494602 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489695    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:19:32.494602 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489699    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:19:32.494602 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489702    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:19:32.495046 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489705    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:19:32.495046 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489708    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:19:32.495046 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489711    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:19:32.495046 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489713    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:19:32.495046 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489716    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:19:32.495046 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489718    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:19:32.495046 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489721    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:19:32.495046 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489724    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:19:32.495046 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489726    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:19:32.495046 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489729    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:19:32.495046 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489732    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:19:32.495046 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489735    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:19:32.495046 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489737    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:19:32.495046 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489740    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:19:32.495046 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489743    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:19:32.495046 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489745    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:19:32.495046 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489748    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:19:32.495046 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489750    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:19:32.495046 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489753    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:19:32.495046 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489756    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:19:32.495528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489758    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:19:32.495528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489761    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:19:32.495528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489764    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:19:32.495528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489767    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:19:32.495528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489770    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:19:32.495528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489772    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:19:32.495528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489775    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:19:32.495528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489778    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:19:32.495528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489781    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:19:32.495528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489784    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:19:32.495528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489786    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:19:32.495528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489789    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:19:32.495528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489792    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:19:32.495528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489795    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:19:32.495528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489798    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:19:32.495528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489800    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:19:32.495528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489803    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:19:32.495528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489805    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:19:32.495528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489808    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:19:32.495996 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489810    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:19:32.495996 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489813    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:19:32.495996 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489816    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:19:32.495996 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489820    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:19:32.495996 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489824    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:19:32.495996 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489827    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:19:32.495996 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489830    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:19:32.495996 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489833    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:19:32.495996 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489835    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:19:32.495996 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489838    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:19:32.495996 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489841    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:19:32.495996 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489843    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:19:32.495996 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489846    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:19:32.495996 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489848    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:19:32.495996 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489851    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:19:32.495996 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489853    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:19:32.495996 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489856    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:19:32.495996 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489859    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:19:32.495996 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489861    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:19:32.495996 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489864    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:19:32.496528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489867    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:19:32.496528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489869    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:19:32.496528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489872    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:19:32.496528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489874    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:19:32.496528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489877    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:19:32.496528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489880    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:19:32.496528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489883    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:19:32.496528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489886    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:19:32.496528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489888    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:19:32.496528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489891    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:19:32.496528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489894    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:19:32.496528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489896    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:19:32.496528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489899    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:19:32.496528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489901    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:19:32.496528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489904    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:19:32.496528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489907    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:19:32.496528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489909    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:19:32.496528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489912    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:19:32.496528 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489916    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:19:32.496969 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489918    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:19:32.496969 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.489943    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:19:32.496969 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489977    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:19:32.496969 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489981    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:19:32.496969 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489985    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:19:32.496969 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489988    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:19:32.496969 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489991    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:19:32.496969 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489994    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:19:32.496969 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489997    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:19:32.496969 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.489999    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:19:32.496969 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490002    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:19:32.496969 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490005    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:19:32.496969 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490008    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:19:32.496969 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490011    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:19:32.496969 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490013    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:19:32.497329 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490016    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:19:32.497329 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490018    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:19:32.497329 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490021    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:19:32.497329 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490024    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:19:32.497329 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490028    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:19:32.497329 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490032    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:19:32.497329 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490035    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:19:32.497329 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490038    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:19:32.497329 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490040    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:19:32.497329 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490043    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:19:32.497329 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490046    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:19:32.497329 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490048    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:19:32.497329 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490051    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:19:32.497329 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490053    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:19:32.497329 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490056    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:19:32.497329 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490058    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:19:32.497329 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490061    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:19:32.497329 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490064    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:19:32.497329 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490067    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:19:32.497329 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490069    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:19:32.497819 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490072    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:19:32.497819 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490075    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:19:32.497819 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490077    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:19:32.497819 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490080    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:19:32.497819 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490083    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:19:32.497819 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490085    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:19:32.497819 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490088    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:19:32.497819 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490090    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:19:32.497819 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490093    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:19:32.497819 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490096    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:19:32.497819 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490100    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:19:32.497819 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490102    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:19:32.497819 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490105    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:19:32.497819 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490107    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:19:32.497819 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490110    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:19:32.497819 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490113    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:19:32.497819 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490115    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:19:32.497819 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490118    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:19:32.497819 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490120    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:19:32.498305 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490123    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:19:32.498305 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490126    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:19:32.498305 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490128    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:19:32.498305 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490131    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:19:32.498305 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490133    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:19:32.498305 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490136    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:19:32.498305 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490138    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:19:32.498305 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490141    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:19:32.498305 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490143    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:19:32.498305 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490146    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:19:32.498305 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490148    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:19:32.498305 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490152    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:19:32.498305 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490155    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:19:32.498305 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490158    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:19:32.498305 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490160    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:19:32.498305 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490163    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:19:32.498305 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490165    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:19:32.498305 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490168    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:19:32.498305 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490171    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:19:32.498305 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490173    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:19:32.498784 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490178    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:19:32.498784 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490181    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:19:32.498784 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490184    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:19:32.498784 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490186    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:19:32.498784 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490189    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:19:32.498784 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490191    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:19:32.498784 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490195    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:19:32.498784 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490199    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:19:32.498784 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490204    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:19:32.498784 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490207    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:19:32.498784 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490210    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:19:32.498784 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490213    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:19:32.498784 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490216    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:19:32.498784 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490218    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:19:32.498784 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490221    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:19:32.498784 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490223    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:19:32.498784 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490226    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:19:32.498784 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490229    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:19:32.498784 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490232    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:19:32.499289 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490234    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:19:32.499289 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490237    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:19:32.499289 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490239    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:19:32.499289 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490242    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:19:32.499289 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490245    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:19:32.499289 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490248    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:19:32.499289 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490252    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:19:32.499289 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490255    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:19:32.499289 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490258    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:19:32.499289 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490260    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:19:32.499289 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490263    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:19:32.499289 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490266    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:19:32.499289 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490268    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:19:32.499289 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490273    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:19:32.499289 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:32.490276    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:19:32.499289 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.490334    2578 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 01:19:32.499289 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.492710    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 01:19:32.499289 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.493688    2578 server.go:1014] "Starting client certificate rotation"
Oct 31 01:19:32.499289 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.493783    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 01:19:32.499748 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.493825    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 01:19:32.499748 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.495928    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 01:19:32.499748 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.497113    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 01:19:32.509537 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.509515    2578 log.go:25] "Validated CRI v1 runtime API"
Oct 31 01:19:32.513495 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.513469    2578 log.go:25] "Validated CRI v1 image API"
Oct 31 01:19:32.514332 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.514305    2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 01:19:32.516961 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.516940    2578 fs.go:135] Filesystem UUIDs: map[3b3b78fb-536b-40e5-8c9c-848963575470:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9cea673a-776a-431e-ad60-8fa157122841:/dev/nvme0n1p4]
Oct 31 01:19:32.517042 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.516959    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 01:19:32.523839 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.523712    2578 manager.go:217] Machine: {Timestamp:2026-10-31 01:19:32.521768785 +0000 UTC m=+0.378655935 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099926 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2082908ab17fa0d0a29ca0b909c15f SystemUUID:ec208290-8ab1-7fa0-d0a2-9ca0b909c15f BootID:5f1a19c8-7324-48a2-8b26-c1e71927a23d Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:dd:cc:0e:59 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:dd:cc:0e:59 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:81:46:b7:56:69 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 01:19:32.523839 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.523829    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 01:19:32.523953 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.523932    2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 01:19:32.525204 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.525176    2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 01:19:32.525347 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.525206    2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-98.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 01:19:32.526266 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.526256    2578 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 01:19:32.526303 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.526269    2578 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 01:19:32.526303 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.526288    2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 01:19:32.526303 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.526302    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 01:19:32.527468 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.527436    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 01:19:32.527564 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.527553    2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 01:19:32.533210 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.533198    2578 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 01:19:32.533349 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.533340    2578 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 01:19:32.533385 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.533373    2578 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 01:19:32.533385 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.533383    2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 01:19:32.533441 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.533393    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 01:19:32.533441 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.533402    2578 kubelet.go:405] "Adding apiserver pod source"
Oct 31 01:19:32.533441 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.533410    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 01:19:32.539271 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.533799    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 01:19:32.540436 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.540417    2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 01:19:32.545693 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.545678    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 01:19:32.546598 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.546585    2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 01:19:32.546683 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.546617    2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 01:19:32.548287 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.548272    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 01:19:32.548357 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.548292    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 01:19:32.548357 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.548298    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 01:19:32.548357 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.548304    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 01:19:32.548357 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.548310    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 01:19:32.548357 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.548316    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 01:19:32.548357 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.548324    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 01:19:32.548357 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.548329    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 01:19:32.548357 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.548338    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 01:19:32.548357 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.548351    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 01:19:32.548646 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.548364    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 01:19:32.548646 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.548374    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 01:19:32.549862 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.549851    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 01:19:32.549896 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.549865    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 01:19:32.553996 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.553976    2578 server.go:1263] "Started kubelet"
Oct 31 01:19:32.554155 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:32.554136    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 01:19:32.554231 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.554189    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 01:19:32.554275 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.554248    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 01:19:32.554491 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.554464    2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 01:19:32.554945 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.554922    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8qgp5"
Oct 31 01:19:32.554938 ip-10-0-133-98 systemd[1]: Started Kubernetes Kubelet.
Oct 31 01:19:32.555189 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.554998    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-98.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 01:19:32.555346 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:32.555317    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-98.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 01:19:32.555786 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.555769    2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 01:19:32.557012 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.556994    2578 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 01:19:32.560837 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:32.560808    2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 01:19:32.563720 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.563688    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 01:19:32.564355 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.564337    2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 01:19:32.564618 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:32.564600    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-98.us-east-2.compute.internal\" not found"
Oct 31 01:19:32.565312 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.565289    2578 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 01:19:32.565312 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.565315    2578 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 01:19:32.565475 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.565290    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 01:19:32.565475 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.565437    2578 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 01:19:32.565475 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.565458    2578 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 01:19:32.566049 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.566031    2578 factory.go:55] Registering systemd factory
Oct 31 01:19:32.566049 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.566051    2578 factory.go:223] Registration of the systemd container factory successfully
Oct 31 01:19:32.566392 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.566377    2578 factory.go:153] Registering CRI-O factory
Oct 31 01:19:32.566482 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.566395    2578 factory.go:223] Registration of the crio container factory successfully
Oct 31 01:19:32.566482 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.566474    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 01:19:32.566586 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.566492    2578 factory.go:103] Registering Raw factory
Oct 31 01:19:32.566586 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.566508    2578 manager.go:1199] Started watching for new ooms in manager
Oct 31 01:19:32.568546 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:32.565727    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-98.us-east-2.compute.internal.18e378811cdcb31f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-98.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-98.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 01:19:32.553929503 +0000 UTC m=+0.410816646,LastTimestamp:2026-10-31 01:19:32.553929503 +0000 UTC m=+0.410816646,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-98.us-east-2.compute.internal,}"
Oct 31 01:19:32.569178 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.569151    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8qgp5"
Oct 31 01:19:32.569258 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:32.569178    2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-98.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 01:19:32.569374 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.569355    2578 manager.go:319] Starting recovery of all containers
Oct 31 01:19:32.573152 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:32.573128    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 31 01:19:32.575839 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.575809    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 01:19:32.581215 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.581195    2578 manager.go:324] Recovery completed
Oct 31 01:19:32.585212 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.585199    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:19:32.587601 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.587585    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-98.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:19:32.587692 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.587622    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-98.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:19:32.587692 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.587638    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-98.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:19:32.588157 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.588143    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:19:32.588224 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.588143    2578 cpu_manager.go:226] "Starting" policy="none"
Oct 31 01:19:32.588224 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.588171    2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 01:19:32.588224 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.588189    2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 01:19:32.588869 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.588854    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-98.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:19:32.588910 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.588884    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-98.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:19:32.588910 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.588897    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-98.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:19:32.590427 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.590414    2578 policy_none.go:50] "Start"
Oct 31 01:19:32.591037 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.591027    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 01:19:32.591069 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.591046    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 01:19:32.593256 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.593246    2578 policy_none.go:44] "Start"
Oct 31 01:19:32.634216 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.634015    2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 01:19:32.638195 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.638176    2578 manager.go:341] "Starting Device Plugin manager"
Oct 31 01:19:32.638998 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:32.638978    2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 01:19:32.638998 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.638999    2578 server.go:87] "Starting device plugin registration server"
Oct 31 01:19:32.639281 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.639267    2578 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 01:19:32.639341 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.639282    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 01:19:32.639397 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.639382    2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 01:19:32.639767 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.639476    2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 01:19:32.639767 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.639493    2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 01:19:32.639985 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:32.639969    2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 01:19:32.640050 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:32.640020    2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-98.us-east-2.compute.internal\" not found"
Oct 31 01:19:32.728468 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.728385    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 01:19:32.728468 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.728411    2578 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 01:19:32.728468 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.728434    2578 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 01:19:32.730151 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:32.728480    2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 01:19:32.732949 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.732931    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:19:32.740001 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.739987    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:19:32.740863 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.740847    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-98.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:19:32.740943 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.740876    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-98.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:19:32.740943 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.740886    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-98.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:19:32.740943 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.740911    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:19:32.750405 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.750388    2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:19:32.750517 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:32.750418    2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-98.us-east-2.compute.internal\": node \"ip-10-0-133-98.us-east-2.compute.internal\" not found"
Oct 31 01:19:32.770346 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:32.770322    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-98.us-east-2.compute.internal\" not found"
Oct 31 01:19:32.829485 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.829420    2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal"]
Oct 31 01:19:32.829622 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.829528    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:19:32.831476 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.831438    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-98.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:19:32.831560 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.831494    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-98.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:19:32.831560 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.831508    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-98.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:19:32.833310 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.833297    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:19:32.833488 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.833472    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:19:32.833561 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.833514    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:19:32.834128 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.834107    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-98.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:19:32.834128 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.834125    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-98.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:19:32.834246 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.834139    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-98.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:19:32.834246 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.834148    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-98.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:19:32.834246 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.834159    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-98.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:19:32.834333 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.834149    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-98.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:19:32.835908 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.835895    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:19:32.835959 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.835918    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:19:32.836714 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.836699    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-98.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:19:32.836788 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.836727    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-98.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:19:32.836788 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.836738    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-98.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:19:32.855335 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:32.855316    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-98.us-east-2.compute.internal\" not found" node="ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:19:32.858194 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:32.858175    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-98.us-east-2.compute.internal\" not found" node="ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:19:32.870496 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:32.870479    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-98.us-east-2.compute.internal\" not found"
Oct 31 01:19:32.966777 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.966745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a2a6228f7ebdca678429d632d3fb70c4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal\" (UID: \"a2a6228f7ebdca678429d632d3fb70c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:19:32.966777 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.966778    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2a6228f7ebdca678429d632d3fb70c4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal\" (UID: \"a2a6228f7ebdca678429d632d3fb70c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:19:32.966914 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:32.966797    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7b4242bdffafd37f633de48cdce1cc09-config\") pod \"kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal\" (UID: \"7b4242bdffafd37f633de48cdce1cc09\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:19:32.970834 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:32.970815    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-98.us-east-2.compute.internal\" not found"
Oct 31 01:19:33.067675 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:33.067618    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a2a6228f7ebdca678429d632d3fb70c4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal\" (UID: \"a2a6228f7ebdca678429d632d3fb70c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:19:33.067675 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:33.067654    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2a6228f7ebdca678429d632d3fb70c4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal\" (UID: \"a2a6228f7ebdca678429d632d3fb70c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:19:33.067675 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:33.067673    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7b4242bdffafd37f633de48cdce1cc09-config\") pod \"kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal\" (UID: \"7b4242bdffafd37f633de48cdce1cc09\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:19:33.067787 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:33.067714    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a2a6228f7ebdca678429d632d3fb70c4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal\" (UID: \"a2a6228f7ebdca678429d632d3fb70c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:19:33.067787 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:33.067714    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2a6228f7ebdca678429d632d3fb70c4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal\" (UID: \"a2a6228f7ebdca678429d632d3fb70c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:19:33.067787 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:33.067726    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7b4242bdffafd37f633de48cdce1cc09-config\") pod \"kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal\" (UID: \"7b4242bdffafd37f633de48cdce1cc09\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:19:33.071701 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:33.071686    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-98.us-east-2.compute.internal\" not found"
Oct 31 01:19:33.161364 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:33.161335    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:19:33.161449 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:33.161420    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:19:33.172506 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:33.172485    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-98.us-east-2.compute.internal\" not found"
Oct 31 01:19:33.273346 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:33.273310    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-98.us-east-2.compute.internal\" not found"
Oct 31 01:19:33.373665 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:33.373578    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-98.us-east-2.compute.internal\" not found"
Oct 31 01:19:33.474070 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:33.474033    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-98.us-east-2.compute.internal\" not found"
Oct 31 01:19:33.493730 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:33.493692    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 01:19:33.493890 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:33.493844    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 01:19:33.517362 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:33.517337    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:19:33.574598 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:33.574569    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-98.us-east-2.compute.internal\" not found"
Oct 31 01:19:33.575759 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:33.575736    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 01:14:33 +0000 UTC" deadline="2028-06-13 04:06:58.692412317 +0000 UTC"
Oct 31 01:19:33.575809 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:33.575760    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14186h47m25.116655455s"
Oct 31 01:19:33.608010 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:33.607978    2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:19:33.675410 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:33.675312    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-98.us-east-2.compute.internal\" not found"
Oct 31 01:19:33.776350 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:33.776307    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-98.us-east-2.compute.internal\" not found"
Oct 31 01:19:33.877121 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:33.877083    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-98.us-east-2.compute.internal\" not found"
Oct 31 01:19:33.918162 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:33.916786    2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:19:33.964961 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:33.964893    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:19:33.973997 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:33.973972    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:19:33.974830 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:33.974814    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:19:33.982452 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:33.982416    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a6228f7ebdca678429d632d3fb70c4.slice/crio-4f82d934690d4d27dd037f936a6e4c17907e23379df72d3eb8ba44e6d2c6d442 WatchSource:0}: Error finding container 4f82d934690d4d27dd037f936a6e4c17907e23379df72d3eb8ba44e6d2c6d442: Status 404 returned error can't find the container with id 4f82d934690d4d27dd037f936a6e4c17907e23379df72d3eb8ba44e6d2c6d442
Oct 31 01:19:33.985621 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:33.985567    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4242bdffafd37f633de48cdce1cc09.slice/crio-167a4eb856c92f9ac3fc51b87c43d6bca9a72a347aae3bf431c7b8203a9de001 WatchSource:0}: Error finding container 167a4eb856c92f9ac3fc51b87c43d6bca9a72a347aae3bf431c7b8203a9de001: Status 404 returned error can't find the container with id 167a4eb856c92f9ac3fc51b87c43d6bca9a72a347aae3bf431c7b8203a9de001
Oct 31 01:19:33.986322 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:33.986294    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:19:33.988635 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:33.988617    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:19:34.535095 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.535065    2578 apiserver.go:52] "Watching apiserver"
Oct 31 01:19:34.541326 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.541300    2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 01:19:34.541622 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.541600    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-lhhnx","openshift-ovn-kubernetes/ovnkube-node-tmh79","kube-system/global-pull-secret-syncer-5crfw","kube-system/konnectivity-agent-wnmfp","openshift-dns/node-resolver-xghmx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-vfk9h","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv","openshift-image-registry/node-ca-tzz5q","openshift-multus/multus-fqpcx","kube-system/kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-m7xdt","openshift-multus/multus-additional-cni-plugins-ll9gg","openshift-network-diagnostics/network-check-target-jn7dx"]
Oct 31 01:19:34.544309 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.544285    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:19:34.544465 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:34.544397    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfk9h" podUID="d9e6beef-017b-49a6-81a3-45b8faf145fc"
Oct 31 01:19:34.546774 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.546754    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:19:34.549178 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.549153    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:34.549296 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:34.549232    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfw" podUID="f8449422-f48d-4df2-8548-9de1c5339fd0"
Oct 31 01:19:34.550210 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.550032    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h7598\""
Oct 31 01:19:34.550210 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.550080    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 01:19:34.550210 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.550086    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 01:19:34.550210 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.550099    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 01:19:34.550210 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.550097    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 01:19:34.550210 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.550184    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 01:19:34.551517 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.551499    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wnmfp"
Oct 31 01:19:34.553879 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.553861    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xghmx"
Oct 31 01:19:34.556281 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.556259    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:19:34.556387 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:34.556353    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn7dx" podUID="2aa13b7e-bcfc-4ca6-96bd-086caf620c9c"
Oct 31 01:19:34.559606 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.559581    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tzz5q"
Oct 31 01:19:34.562713 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.562683    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lhhnx"
Oct 31 01:19:34.564552 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.564532    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 01:19:34.565561 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.565541    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.568340 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.568309    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.571303 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.571278    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:19:34.574583 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.574561    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.577238 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.577208    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 01:19:34.578175 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.578155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81c10625-0368-49c7-89cf-eb51f06005d0-cni-binary-copy\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.578258 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.578183    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ec3e19fc-4890-424b-86ef-f5ad1e926158-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:19:34.578258 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.578208    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-systemd-units\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.578347 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.578235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b7841e1-63bb-4e71-972e-1020c47d2ccb-ovnkube-script-lib\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.578347 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.578294    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9jkq\" (UniqueName: \"kubernetes.io/projected/86947040-78ac-43d5-b932-9acd63cf43e7-kube-api-access-v9jkq\") pod \"node-resolver-xghmx\" (UID: \"86947040-78ac-43d5-b932-9acd63cf43e7\") " pod="openshift-dns/node-resolver-xghmx"
Oct 31 01:19:34.578452 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.578416    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr95h\" (UniqueName: \"kubernetes.io/projected/ec3e19fc-4890-424b-86ef-f5ad1e926158-kube-api-access-qr95h\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:19:34.578520 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.578485    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f-konnectivity-ca\") pod \"konnectivity-agent-wnmfp\" (UID: \"308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f\") " pod="kube-system/konnectivity-agent-wnmfp"
Oct 31 01:19:34.578580 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.578560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xs45\" (UniqueName: \"kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45\") pod \"network-check-target-jn7dx\" (UID: \"2aa13b7e-bcfc-4ca6-96bd-086caf620c9c\") " pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:19:34.578655 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.578631    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec3e19fc-4890-424b-86ef-f5ad1e926158-cnibin\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:19:34.578715 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.578685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f8449422-f48d-4df2-8548-9de1c5339fd0-dbus\") pod \"global-pull-secret-syncer-5crfw\" (UID: \"f8449422-f48d-4df2-8548-9de1c5339fd0\") " pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:34.578771 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.578744    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db94c614-bfe5-4328-b1aa-770694e4fca1-host-slash\") pod \"iptables-alerter-lhhnx\" (UID: \"db94c614-bfe5-4328-b1aa-770694e4fca1\") " pod="openshift-network-operator/iptables-alerter-lhhnx"
Oct 31 01:19:34.578817 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.578779    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-run-k8s-cni-cncf-io\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.578869 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.578814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-run-netns\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.578869 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.578858    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-run-ovn-kubernetes\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.578976 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.578878    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b7841e1-63bb-4e71-972e-1020c47d2ccb-ovn-node-metrics-cert\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.578976 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.578962    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-cni-netd\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.579082 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579013    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec3e19fc-4890-424b-86ef-f5ad1e926158-cni-binary-copy\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:19:34.579082 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579045    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec3e19fc-4890-424b-86ef-f5ad1e926158-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:19:34.579082 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579072    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-var-lib-openvswitch\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.579210 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-etc-openvswitch\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.579210 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579124    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-run-ovn\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.579210 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579181    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwd6x\" (UniqueName: \"kubernetes.io/projected/db94c614-bfe5-4328-b1aa-770694e4fca1-kube-api-access-xwd6x\") pod \"iptables-alerter-lhhnx\" (UID: \"db94c614-bfe5-4328-b1aa-770694e4fca1\") " pod="openshift-network-operator/iptables-alerter-lhhnx"
Oct 31 01:19:34.579369 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-run-openvswitch\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.579369 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579272    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-node-log\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.579369 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579304    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/86947040-78ac-43d5-b932-9acd63cf43e7-hosts-file\") pod \"node-resolver-xghmx\" (UID: \"86947040-78ac-43d5-b932-9acd63cf43e7\") " pod="openshift-dns/node-resolver-xghmx"
Oct 31 01:19:34.579369 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579334    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-multus-conf-dir\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.579602 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579360    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b7841e1-63bb-4e71-972e-1020c47d2ccb-ovnkube-config\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.579602 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579402    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-system-cni-dir\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.579602 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579428    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-multus-socket-dir-parent\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.579602 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579444    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-etc-kubernetes\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.579602 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579485    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f8449422-f48d-4df2-8548-9de1c5339fd0-kubelet-config\") pod \"global-pull-secret-syncer-5crfw\" (UID: \"f8449422-f48d-4df2-8548-9de1c5339fd0\") " pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:34.579602 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579509    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-run-netns\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.579602 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579533    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-log-socket\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.579602 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579576    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/52e25fa9-6c04-453a-a0b2-cca32a428a5a-serviceca\") pod \"node-ca-tzz5q\" (UID: \"52e25fa9-6c04-453a-a0b2-cca32a428a5a\") " pod="openshift-image-registry/node-ca-tzz5q"
Oct 31 01:19:34.579602 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579596    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8tn8\" (UniqueName: \"kubernetes.io/projected/81c10625-0368-49c7-89cf-eb51f06005d0-kube-api-access-v8tn8\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.580036 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579614    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec3e19fc-4890-424b-86ef-f5ad1e926158-system-cni-dir\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:19:34.580036 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f-agent-certs\") pod \"konnectivity-agent-wnmfp\" (UID: \"308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f\") " pod="kube-system/konnectivity-agent-wnmfp"
Oct 31 01:19:34.580036 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579688    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-kubelet\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.580036 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b7841e1-63bb-4e71-972e-1020c47d2ccb-env-overrides\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.580036 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579774    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52e25fa9-6c04-453a-a0b2-cca32a428a5a-host\") pod \"node-ca-tzz5q\" (UID: \"52e25fa9-6c04-453a-a0b2-cca32a428a5a\") " pod="openshift-image-registry/node-ca-tzz5q"
Oct 31 01:19:34.580036 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579802    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw5bx\" (UniqueName: \"kubernetes.io/projected/52e25fa9-6c04-453a-a0b2-cca32a428a5a-kube-api-access-nw5bx\") pod \"node-ca-tzz5q\" (UID: \"52e25fa9-6c04-453a-a0b2-cca32a428a5a\") " pod="openshift-image-registry/node-ca-tzz5q"
Oct 31 01:19:34.580036 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579847    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5md27\" (UniqueName: \"kubernetes.io/projected/d9e6beef-017b-49a6-81a3-45b8faf145fc-kube-api-access-5md27\") pod \"network-metrics-daemon-vfk9h\" (UID: \"d9e6beef-017b-49a6-81a3-45b8faf145fc\") " pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:19:34.580036 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579880    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh6l2\" (UniqueName: \"kubernetes.io/projected/6b7841e1-63bb-4e71-972e-1020c47d2ccb-kube-api-access-kh6l2\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.580036 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579920    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs\") pod \"network-metrics-daemon-vfk9h\" (UID: \"d9e6beef-017b-49a6-81a3-45b8faf145fc\") " pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:19:34.580036 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.579947    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-slash\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.580036 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.580012    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.580463 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.580113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-var-lib-cni-bin\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.580463 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.580144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-var-lib-cni-multus\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.580463 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.580168    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-hostroot\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.580463 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.580192    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/81c10625-0368-49c7-89cf-eb51f06005d0-multus-daemon-config\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.580463 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.580216    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-run-multus-certs\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.580463 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.580241    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db94c614-bfe5-4328-b1aa-770694e4fca1-iptables-alerter-script\") pod \"iptables-alerter-lhhnx\" (UID: \"db94c614-bfe5-4328-b1aa-770694e4fca1\") " pod="openshift-network-operator/iptables-alerter-lhhnx"
Oct 31 01:19:34.580463 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.580267    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret\") pod \"global-pull-secret-syncer-5crfw\" (UID: \"f8449422-f48d-4df2-8548-9de1c5339fd0\") " pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:34.580463 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.580290    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-run-systemd\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.580463 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.580314    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-multus-cni-dir\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.580463 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.580338    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-cnibin\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.580463 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.580364    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ec3e19fc-4890-424b-86ef-f5ad1e926158-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:19:34.580463 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.580390    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86947040-78ac-43d5-b932-9acd63cf43e7-tmp-dir\") pod \"node-resolver-xghmx\" (UID: \"86947040-78ac-43d5-b932-9acd63cf43e7\") " pod="openshift-dns/node-resolver-xghmx"
Oct 31 01:19:34.580463 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.580417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-var-lib-kubelet\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.580463 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.580450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-cni-bin\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.603473 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.603442    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gx9b8"
Oct 31 01:19:34.613890 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.613848    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gx9b8"
Oct 31 01:19:34.659424 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.659378    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 01:19:34.659424 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.659423    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 01:19:34.659853 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.659830    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 01:19:34.659976 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.659956    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-82mbc\""
Oct 31 01:19:34.660038 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.660017    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 01:19:34.660038 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.659877    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 01:19:34.660158 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.660078    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 01:19:34.660217 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.659844    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 01:19:34.660751 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.660318    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 01:19:34.660751 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.660341    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hm2lp\""
Oct 31 01:19:34.660751 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.660379    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 01:19:34.660751 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.660323    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 01:19:34.660751 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.660379    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 01:19:34.660751 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.660388    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 01:19:34.660751 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.660413    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fnh8f\""
Oct 31 01:19:34.660751 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.660420    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 01:19:34.660751 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.660413    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 01:19:34.660751 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.660444    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 01:19:34.660751 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.660418    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 01:19:34.660751 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.660415    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 01:19:34.660751 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.660390    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 01:19:34.660751 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.660476    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 01:19:34.660751 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.660490    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:19:34.660751 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.660498    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vjbgx\""
Oct 31 01:19:34.660751 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.660490    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:19:34.660751 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.660521    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tsbmr\""
Oct 31 01:19:34.660751 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.660552    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vj55m\""
Oct 31 01:19:34.660751 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.660552    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gv7wl\""
Oct 31 01:19:34.660751 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.660597    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d5hw4\""
Oct 31 01:19:34.660751 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.660723    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 01:19:34.666245 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.666222    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 01:19:34.678153 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.678122    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:19:34.681434 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.681407    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec3e19fc-4890-424b-86ef-f5ad1e926158-cni-binary-copy\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:19:34.681560 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.681448    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec3e19fc-4890-424b-86ef-f5ad1e926158-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:19:34.681631 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.681602    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-var-lib-openvswitch\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.681686 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.681634    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-etc-openvswitch\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.681686 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.681661    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-run-ovn\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.681805 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.681683    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec3e19fc-4890-424b-86ef-f5ad1e926158-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:19:34.681805 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.681723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-sys-fs\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:19:34.681805 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.681745    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-etc-openvswitch\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.681805 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.681760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwd6x\" (UniqueName: \"kubernetes.io/projected/db94c614-bfe5-4328-b1aa-770694e4fca1-kube-api-access-xwd6x\") pod \"iptables-alerter-lhhnx\" (UID: \"db94c614-bfe5-4328-b1aa-770694e4fca1\") " pod="openshift-network-operator/iptables-alerter-lhhnx"
Oct 31 01:19:34.682043 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.681805    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-var-lib-openvswitch\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.682043 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.681825    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-run-openvswitch\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.682043 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.681849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-node-log\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.682043 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.681884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/86947040-78ac-43d5-b932-9acd63cf43e7-hosts-file\") pod \"node-resolver-xghmx\" (UID: \"86947040-78ac-43d5-b932-9acd63cf43e7\") " pod="openshift-dns/node-resolver-xghmx"
Oct 31 01:19:34.682043 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.681911    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-multus-conf-dir\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.682043 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.681956    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b7841e1-63bb-4e71-972e-1020c47d2ccb-ovnkube-config\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.682043 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.681983    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-registration-dir\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:19:34.682043 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.682008    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-systemd\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.682043 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.682029    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec3e19fc-4890-424b-86ef-f5ad1e926158-cni-binary-copy\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:19:34.682043 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.682033    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-system-cni-dir\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.682556 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.682079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-multus-socket-dir-parent\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.682556 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.682104    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-etc-kubernetes\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.682556 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.682128    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f8449422-f48d-4df2-8548-9de1c5339fd0-kubelet-config\") pod \"global-pull-secret-syncer-5crfw\" (UID: \"f8449422-f48d-4df2-8548-9de1c5339fd0\") " pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:34.682556 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.682155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-run-netns\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.682556 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.682178    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-log-socket\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.682556 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.682218    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-multus-conf-dir\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.686619 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.682208    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-kubernetes\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.686619 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.685559    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/52e25fa9-6c04-453a-a0b2-cca32a428a5a-serviceca\") pod \"node-ca-tzz5q\" (UID: \"52e25fa9-6c04-453a-a0b2-cca32a428a5a\") " pod="openshift-image-registry/node-ca-tzz5q"
Oct 31 01:19:34.686619 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.685611    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8tn8\" (UniqueName: \"kubernetes.io/projected/81c10625-0368-49c7-89cf-eb51f06005d0-kube-api-access-v8tn8\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.686619 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.685660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec3e19fc-4890-424b-86ef-f5ad1e926158-system-cni-dir\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:19:34.686619 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.685702    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-host\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.686619 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.685734    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmwzw\" (UniqueName: \"kubernetes.io/projected/b0cbe408-97f0-48da-ab7d-ab4d144f0796-kube-api-access-xmwzw\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.686619 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.685771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f-agent-certs\") pod \"konnectivity-agent-wnmfp\" (UID: \"308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f\") " pod="kube-system/konnectivity-agent-wnmfp"
Oct 31 01:19:34.686619 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.685809    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-kubelet\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.686619 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.685841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b7841e1-63bb-4e71-972e-1020c47d2ccb-env-overrides\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.686619 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.685900    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec3e19fc-4890-424b-86ef-f5ad1e926158-system-cni-dir\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:19:34.686619 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.685903    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-run-ovn\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.686619 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.686091    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-kubelet\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.686619 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.686161    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-node-log\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.686619 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.686218    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-run-openvswitch\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.686619 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.686241    2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 01:19:34.686619 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.686287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:19:34.686619 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.686350    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-sysctl-d\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.686619 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.686428    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b7841e1-63bb-4e71-972e-1020c47d2ccb-env-overrides\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.687427 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.686453    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f8449422-f48d-4df2-8548-9de1c5339fd0-kubelet-config\") pod \"global-pull-secret-syncer-5crfw\" (UID: \"f8449422-f48d-4df2-8548-9de1c5339fd0\") " pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:34.687427 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.686513    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-etc-kubernetes\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.687427 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.686548    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-multus-socket-dir-parent\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.687427 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.686624    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52e25fa9-6c04-453a-a0b2-cca32a428a5a-host\") pod \"node-ca-tzz5q\" (UID: \"52e25fa9-6c04-453a-a0b2-cca32a428a5a\") " pod="openshift-image-registry/node-ca-tzz5q"
Oct 31 01:19:34.687427 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.686705    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw5bx\" (UniqueName: \"kubernetes.io/projected/52e25fa9-6c04-453a-a0b2-cca32a428a5a-kube-api-access-nw5bx\") pod \"node-ca-tzz5q\" (UID: \"52e25fa9-6c04-453a-a0b2-cca32a428a5a\") " pod="openshift-image-registry/node-ca-tzz5q"
Oct 31 01:19:34.687427 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.686725    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/86947040-78ac-43d5-b932-9acd63cf43e7-hosts-file\") pod \"node-resolver-xghmx\" (UID: \"86947040-78ac-43d5-b932-9acd63cf43e7\") " pod="openshift-dns/node-resolver-xghmx"
Oct 31 01:19:34.687427 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.686749    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52e25fa9-6c04-453a-a0b2-cca32a428a5a-host\") pod \"node-ca-tzz5q\" (UID: \"52e25fa9-6c04-453a-a0b2-cca32a428a5a\") " pod="openshift-image-registry/node-ca-tzz5q"
Oct 31 01:19:34.687427 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.686806    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5md27\" (UniqueName: \"kubernetes.io/projected/d9e6beef-017b-49a6-81a3-45b8faf145fc-kube-api-access-5md27\") pod \"network-metrics-daemon-vfk9h\" (UID: \"d9e6beef-017b-49a6-81a3-45b8faf145fc\") " pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:19:34.687427 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.686842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh6l2\" (UniqueName: \"kubernetes.io/projected/6b7841e1-63bb-4e71-972e-1020c47d2ccb-kube-api-access-kh6l2\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.687427 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.686901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs\") pod \"network-metrics-daemon-vfk9h\" (UID: \"d9e6beef-017b-49a6-81a3-45b8faf145fc\") " pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:19:34.687427 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.686928    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-slash\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.687427 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.686961    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-run\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.687427 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.686993    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-tuned\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.687427 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.687067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.687427 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.687103    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-var-lib-cni-bin\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.687427 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.687146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-var-lib-cni-multus\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.687427 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.687177    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-hostroot\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.687427 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.687209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/81c10625-0368-49c7-89cf-eb51f06005d0-multus-daemon-config\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.688320 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.687234    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-run-multus-certs\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.688320 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.687263    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b7841e1-63bb-4e71-972e-1020c47d2ccb-ovnkube-config\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.688320 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.687268    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db94c614-bfe5-4328-b1aa-770694e4fca1-iptables-alerter-script\") pod \"iptables-alerter-lhhnx\" (UID: \"db94c614-bfe5-4328-b1aa-770694e4fca1\") " pod="openshift-network-operator/iptables-alerter-lhhnx"
Oct 31 01:19:34.688320 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.687335    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-run-netns\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.688320 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.687387    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-log-socket\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.688320 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.687443    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-system-cni-dir\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.688320 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.687611    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret\") pod \"global-pull-secret-syncer-5crfw\" (UID: \"f8449422-f48d-4df2-8548-9de1c5339fd0\") " pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:34.688320 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.687646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-run-systemd\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.688320 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.687685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0cbe408-97f0-48da-ab7d-ab4d144f0796-tmp\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.688320 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.687724    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-multus-cni-dir\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.688320 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.687757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-cnibin\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.688320 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.687792    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ec3e19fc-4890-424b-86ef-f5ad1e926158-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:19:34.688320 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.687810    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db94c614-bfe5-4328-b1aa-770694e4fca1-iptables-alerter-script\") pod \"iptables-alerter-lhhnx\" (UID: \"db94c614-bfe5-4328-b1aa-770694e4fca1\") " pod="openshift-network-operator/iptables-alerter-lhhnx"
Oct 31 01:19:34.688320 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.687824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86947040-78ac-43d5-b932-9acd63cf43e7-tmp-dir\") pod \"node-resolver-xghmx\" (UID: \"86947040-78ac-43d5-b932-9acd63cf43e7\") " pod="openshift-dns/node-resolver-xghmx"
Oct 31 01:19:34.688320 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.687873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-var-lib-kubelet\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.688320 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.687916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-cni-bin\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.688320 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.687885    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-var-lib-cni-bin\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.688320 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.688179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81c10625-0368-49c7-89cf-eb51f06005d0-cni-binary-copy\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.689219 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.688222    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ec3e19fc-4890-424b-86ef-f5ad1e926158-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:19:34.689219 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.688253    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-systemd-units\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.689219 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.688287    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b7841e1-63bb-4e71-972e-1020c47d2ccb-ovnkube-script-lib\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.689219 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.688319    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9jkq\" (UniqueName: \"kubernetes.io/projected/86947040-78ac-43d5-b932-9acd63cf43e7-kube-api-access-v9jkq\") pod \"node-resolver-xghmx\" (UID: \"86947040-78ac-43d5-b932-9acd63cf43e7\") " pod="openshift-dns/node-resolver-xghmx"
Oct 31 01:19:34.689219 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.688352    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr95h\" (UniqueName: \"kubernetes.io/projected/ec3e19fc-4890-424b-86ef-f5ad1e926158-kube-api-access-qr95h\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:19:34.689219 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.688384    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f-konnectivity-ca\") pod \"konnectivity-agent-wnmfp\" (UID: \"308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f\") " pod="kube-system/konnectivity-agent-wnmfp"
Oct 31 01:19:34.689219 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.688412    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xs45\" (UniqueName: \"kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45\") pod \"network-check-target-jn7dx\" (UID: \"2aa13b7e-bcfc-4ca6-96bd-086caf620c9c\") " pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:19:34.689219 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.688447    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:19:34.689219 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.688480    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-sys\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.689219 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.688515    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-lib-modules\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.689219 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.688557    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/52e25fa9-6c04-453a-a0b2-cca32a428a5a-serviceca\") pod \"node-ca-tzz5q\" (UID: \"52e25fa9-6c04-453a-a0b2-cca32a428a5a\") " pod="openshift-image-registry/node-ca-tzz5q"
Oct 31 01:19:34.689219 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.688571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-var-lib-kubelet\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.689219 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.688632    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-slash\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.689219 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:34.688666    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:19:34.689219 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.688674    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-var-lib-cni-multus\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.689219 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.688710    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.690040 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:34.688765    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs podName:d9e6beef-017b-49a6-81a3-45b8faf145fc nodeName:}" failed. No retries permitted until 2026-10-31 01:19:35.188724659 +0000 UTC m=+3.045611818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs") pod "network-metrics-daemon-vfk9h" (UID: "d9e6beef-017b-49a6-81a3-45b8faf145fc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:19:34.690040 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.689008    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-cni-bin\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.690040 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.689803    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ec3e19fc-4890-424b-86ef-f5ad1e926158-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:19:34.690209 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.690098    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-cnibin\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.690209 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.690146    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-hostroot\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.690209 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.690197    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-run-multus-certs\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.690350 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.690283    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b7841e1-63bb-4e71-972e-1020c47d2ccb-ovnkube-script-lib\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.690408 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.690359    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-systemd-units\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.690461 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.690434    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f-konnectivity-ca\") pod \"konnectivity-agent-wnmfp\" (UID: \"308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f\") " pod="kube-system/konnectivity-agent-wnmfp"
Oct 31 01:19:34.690461 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.690436    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-run-systemd\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.690577 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.690554    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec3e19fc-4890-424b-86ef-f5ad1e926158-cnibin\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:19:34.690886 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.690640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f8449422-f48d-4df2-8548-9de1c5339fd0-dbus\") pod \"global-pull-secret-syncer-5crfw\" (UID: \"f8449422-f48d-4df2-8548-9de1c5339fd0\") " pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:34.690886 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.690706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db94c614-bfe5-4328-b1aa-770694e4fca1-host-slash\") pod \"iptables-alerter-lhhnx\" (UID: \"db94c614-bfe5-4328-b1aa-770694e4fca1\") " pod="openshift-network-operator/iptables-alerter-lhhnx"
Oct 31 01:19:34.690886 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.690746    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-socket-dir\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:19:34.690886 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.690782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-device-dir\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:19:34.690886 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:34.690814    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:19:34.691217 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.691007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-modprobe-d\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.691217 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.691040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-sysconfig\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.691217 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.691042    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec3e19fc-4890-424b-86ef-f5ad1e926158-cnibin\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:19:34.691217 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.691075    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-run-k8s-cni-cncf-io\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.691217 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.691108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-run-netns\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.691217 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:34.691132    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret podName:f8449422-f48d-4df2-8548-9de1c5339fd0 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:35.191106695 +0000 UTC m=+3.047993849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret") pod "global-pull-secret-syncer-5crfw" (UID: "f8449422-f48d-4df2-8548-9de1c5339fd0") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:19:34.691217 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.691154    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-run-netns\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.691217 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.691174    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-run-ovn-kubernetes\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.691217 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.691214    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-run-ovn-kubernetes\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.691743 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.691248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b7841e1-63bb-4e71-972e-1020c47d2ccb-ovn-node-metrics-cert\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.691743 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.691284    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-multus-cni-dir\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.691743 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.691290    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f8449422-f48d-4df2-8548-9de1c5339fd0-dbus\") pod \"global-pull-secret-syncer-5crfw\" (UID: \"f8449422-f48d-4df2-8548-9de1c5339fd0\") " pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:34.691743 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.691318    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v74ww\" (UniqueName: \"kubernetes.io/projected/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-kube-api-access-v74ww\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:19:34.691743 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.691341    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db94c614-bfe5-4328-b1aa-770694e4fca1-host-slash\") pod \"iptables-alerter-lhhnx\" (UID: \"db94c614-bfe5-4328-b1aa-770694e4fca1\") " pod="openshift-network-operator/iptables-alerter-lhhnx"
Oct 31 01:19:34.691743 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.691401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-sysctl-conf\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.691743 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.691437    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-run-k8s-cni-cncf-io\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.691743 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.691501    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-cni-netd\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.691743 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.691631    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-cni-netd\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.692193 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.691789    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81c10625-0368-49c7-89cf-eb51f06005d0-cni-binary-copy\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.692193 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.691856    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-var-lib-kubelet\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.692193 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.691980    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/81c10625-0368-49c7-89cf-eb51f06005d0-multus-daemon-config\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.692193 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.692026    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f-agent-certs\") pod \"konnectivity-agent-wnmfp\" (UID: \"308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f\") " pod="kube-system/konnectivity-agent-wnmfp"
Oct 31 01:19:34.692193 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.692139    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86947040-78ac-43d5-b932-9acd63cf43e7-tmp-dir\") pod \"node-resolver-xghmx\" (UID: \"86947040-78ac-43d5-b932-9acd63cf43e7\") " pod="openshift-dns/node-resolver-xghmx"
Oct 31 01:19:34.692193 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.692175    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ec3e19fc-4890-424b-86ef-f5ad1e926158-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:19:34.696248 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.696209    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b7841e1-63bb-4e71-972e-1020c47d2ccb-ovn-node-metrics-cert\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.701941 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.701911    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8tn8\" (UniqueName: \"kubernetes.io/projected/81c10625-0368-49c7-89cf-eb51f06005d0-kube-api-access-v8tn8\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.703428 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:34.703117    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:19:34.703428 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:34.703141    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:19:34.703428 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:34.703154    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9xs45 for pod openshift-network-diagnostics/network-check-target-jn7dx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:19:34.703428 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:34.703217    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45 podName:2aa13b7e-bcfc-4ca6-96bd-086caf620c9c nodeName:}" failed. No retries permitted until 2026-10-31 01:19:35.203197748 +0000 UTC m=+3.060084882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9xs45" (UniqueName: "kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45") pod "network-check-target-jn7dx" (UID: "2aa13b7e-bcfc-4ca6-96bd-086caf620c9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:19:34.704136 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.704106    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9jkq\" (UniqueName: \"kubernetes.io/projected/86947040-78ac-43d5-b932-9acd63cf43e7-kube-api-access-v9jkq\") pod \"node-resolver-xghmx\" (UID: \"86947040-78ac-43d5-b932-9acd63cf43e7\") " pod="openshift-dns/node-resolver-xghmx"
Oct 31 01:19:34.704867 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.704798    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr95h\" (UniqueName: \"kubernetes.io/projected/ec3e19fc-4890-424b-86ef-f5ad1e926158-kube-api-access-qr95h\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:19:34.704867 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.704828    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwd6x\" (UniqueName: \"kubernetes.io/projected/db94c614-bfe5-4328-b1aa-770694e4fca1-kube-api-access-xwd6x\") pod \"iptables-alerter-lhhnx\" (UID: \"db94c614-bfe5-4328-b1aa-770694e4fca1\") " pod="openshift-network-operator/iptables-alerter-lhhnx"
Oct 31 01:19:34.705901 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.705881    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw5bx\" (UniqueName: \"kubernetes.io/projected/52e25fa9-6c04-453a-a0b2-cca32a428a5a-kube-api-access-nw5bx\") pod \"node-ca-tzz5q\" (UID: \"52e25fa9-6c04-453a-a0b2-cca32a428a5a\") " pod="openshift-image-registry/node-ca-tzz5q"
Oct 31 01:19:34.706489 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.706445    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5md27\" (UniqueName: \"kubernetes.io/projected/d9e6beef-017b-49a6-81a3-45b8faf145fc-kube-api-access-5md27\") pod \"network-metrics-daemon-vfk9h\" (UID: \"d9e6beef-017b-49a6-81a3-45b8faf145fc\") " pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:19:34.707536 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.707475    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh6l2\" (UniqueName: \"kubernetes.io/projected/6b7841e1-63bb-4e71-972e-1020c47d2ccb-kube-api-access-kh6l2\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.737960 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.737899    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal" event={"ID":"7b4242bdffafd37f633de48cdce1cc09","Type":"ContainerStarted","Data":"167a4eb856c92f9ac3fc51b87c43d6bca9a72a347aae3bf431c7b8203a9de001"}
Oct 31 01:19:34.739830 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.739788    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal" event={"ID":"a2a6228f7ebdca678429d632d3fb70c4","Type":"ContainerStarted","Data":"4f82d934690d4d27dd037f936a6e4c17907e23379df72d3eb8ba44e6d2c6d442"}
Oct 31 01:19:34.793822 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.792424    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-run\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.793822 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.792499    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-tuned\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.793822 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.792541    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0cbe408-97f0-48da-ab7d-ab4d144f0796-tmp\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.793822 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.792587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:19:34.793822 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.792614    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-sys\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.793822 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.792638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-lib-modules\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.793822 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.792662    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-var-lib-kubelet\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.793822 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.792700    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-socket-dir\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:19:34.793822 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.792725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-device-dir\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:19:34.793822 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.792751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-modprobe-d\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.793822 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.792775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-sysconfig\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.793822 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.792808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v74ww\" (UniqueName: \"kubernetes.io/projected/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-kube-api-access-v74ww\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:19:34.793822 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.792834    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-sysctl-conf\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.793822 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.792864    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-sys-fs\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:19:34.793822 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.792899    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-registration-dir\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:19:34.793822 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.792925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-systemd\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.793822 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.792959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-kubernetes\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.794802 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.792986    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-host\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.794802 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.793010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmwzw\" (UniqueName: \"kubernetes.io/projected/b0cbe408-97f0-48da-ab7d-ab4d144f0796-kube-api-access-xmwzw\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.794802 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.793040    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:19:34.794802 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.793070    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-sysctl-d\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.794802 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.793271    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-sysctl-d\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.794802 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.793354    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-run\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.795820 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.795242    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-device-dir\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:19:34.795820 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.795260    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-systemd\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.795820 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.795260    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-sysconfig\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.795820 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.795346    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-sys-fs\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:19:34.795820 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.795355    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:19:34.795820 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.795404    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-registration-dir\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:19:34.795820 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.795411    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-lib-modules\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.795820 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.795403    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-sysctl-conf\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.795820 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.795472    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-sys\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.795820 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.795495    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-kubernetes\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.795820 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.795534    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-host\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.795820 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.795550    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:19:34.795820 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.795559    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-socket-dir\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:19:34.795820 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.795618    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-var-lib-kubelet\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.795820 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.795795    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-modprobe-d\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.798737 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.798710    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0cbe408-97f0-48da-ab7d-ab4d144f0796-tmp\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.799399 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.799365    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-tuned\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.814480 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.814449    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v74ww\" (UniqueName: \"kubernetes.io/projected/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-kube-api-access-v74ww\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:19:34.815822 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.815802    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmwzw\" (UniqueName: \"kubernetes.io/projected/b0cbe408-97f0-48da-ab7d-ab4d144f0796-kube-api-access-xmwzw\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.878714 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.878679    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wnmfp"
Oct 31 01:19:34.883423 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.883396    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:19:34.886185 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.886157    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xghmx"
Oct 31 01:19:34.892658 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:34.892626    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod308d4c76_fad6_4dfd_8f55_5c5ac0d9b50f.slice/crio-9849baee4ffe03763bf3fa9a1bc91f2b9d3a2c828d6cb0d18680c50d8505efa7 WatchSource:0}: Error finding container 9849baee4ffe03763bf3fa9a1bc91f2b9d3a2c828d6cb0d18680c50d8505efa7: Status 404 returned error can't find the container with id 9849baee4ffe03763bf3fa9a1bc91f2b9d3a2c828d6cb0d18680c50d8505efa7
Oct 31 01:19:34.895410 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:34.895382    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec3e19fc_4890_424b_86ef_f5ad1e926158.slice/crio-f62e7b940902c55bf18ce7cbe90f35be40ea9216ce7ade5ee556fcac904d1e55 WatchSource:0}: Error finding container f62e7b940902c55bf18ce7cbe90f35be40ea9216ce7ade5ee556fcac904d1e55: Status 404 returned error can't find the container with id f62e7b940902c55bf18ce7cbe90f35be40ea9216ce7ade5ee556fcac904d1e55
Oct 31 01:19:34.896404 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.896350    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:19:34.899223 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.899195    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tzz5q"
Oct 31 01:19:34.909229 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:34.909203    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86947040_78ac_43d5_b932_9acd63cf43e7.slice/crio-2a7902ff147c883af26f277241b0fa036f5e800913f8d7aeb1709878f1d15002 WatchSource:0}: Error finding container 2a7902ff147c883af26f277241b0fa036f5e800913f8d7aeb1709878f1d15002: Status 404 returned error can't find the container with id 2a7902ff147c883af26f277241b0fa036f5e800913f8d7aeb1709878f1d15002
Oct 31 01:19:34.910180 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.910157    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lhhnx"
Oct 31 01:19:34.913458 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:34.913431    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c05e0c_b77a_45e4_b49b_1bed7e7588d0.slice/crio-0fd8f4b05e04837e0e0b26e5608e81495837e6095d72ec6fad2bef34dcaf7974 WatchSource:0}: Error finding container 0fd8f4b05e04837e0e0b26e5608e81495837e6095d72ec6fad2bef34dcaf7974: Status 404 returned error can't find the container with id 0fd8f4b05e04837e0e0b26e5608e81495837e6095d72ec6fad2bef34dcaf7974
Oct 31 01:19:34.916587 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:34.916555    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e25fa9_6c04_453a_a0b2_cca32a428a5a.slice/crio-b5d3ecbcd60a975ff2c124eb66e9c9f22998bce80d5d69ad3359575bf6360e19 WatchSource:0}: Error finding container b5d3ecbcd60a975ff2c124eb66e9c9f22998bce80d5d69ad3359575bf6360e19: Status 404 returned error can't find the container with id b5d3ecbcd60a975ff2c124eb66e9c9f22998bce80d5d69ad3359575bf6360e19
Oct 31 01:19:34.922031 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:34.922004    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb94c614_bfe5_4328_b1aa_770694e4fca1.slice/crio-801d38c689ae4e6ea6e4a69f864ddc10453c1012b70f6632a1dfe0a7496e7659 WatchSource:0}: Error finding container 801d38c689ae4e6ea6e4a69f864ddc10453c1012b70f6632a1dfe0a7496e7659: Status 404 returned error can't find the container with id 801d38c689ae4e6ea6e4a69f864ddc10453c1012b70f6632a1dfe0a7496e7659
Oct 31 01:19:34.930958 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.930936    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fqpcx"
Oct 31 01:19:34.933073 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.933035    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:19:34.942366 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:34.942341    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:34.942982 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:34.942953    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c10625_0368_49c7_89cf_eb51f06005d0.slice/crio-5da57e3fdb12e9c5a9f6d66de998f92b009e424cd0f4f55a19e8644e439d3c93 WatchSource:0}: Error finding container 5da57e3fdb12e9c5a9f6d66de998f92b009e424cd0f4f55a19e8644e439d3c93: Status 404 returned error can't find the container with id 5da57e3fdb12e9c5a9f6d66de998f92b009e424cd0f4f55a19e8644e439d3c93
Oct 31 01:19:34.948870 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:34.947280    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0cbe408_97f0_48da_ab7d_ab4d144f0796.slice/crio-00643bc5ed3a191fc78de2b4eac3d3c4daab22f1bd6907b5e381adc3285f7d74 WatchSource:0}: Error finding container 00643bc5ed3a191fc78de2b4eac3d3c4daab22f1bd6907b5e381adc3285f7d74: Status 404 returned error can't find the container with id 00643bc5ed3a191fc78de2b4eac3d3c4daab22f1bd6907b5e381adc3285f7d74
Oct 31 01:19:34.954074 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:19:34.954048    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7841e1_63bb_4e71_972e_1020c47d2ccb.slice/crio-1bc364fd433ed3d2f63c9112e2d37ed896872d6d972a9b39d31c407825d5bd70 WatchSource:0}: Error finding container 1bc364fd433ed3d2f63c9112e2d37ed896872d6d972a9b39d31c407825d5bd70: Status 404 returned error can't find the container with id 1bc364fd433ed3d2f63c9112e2d37ed896872d6d972a9b39d31c407825d5bd70
Oct 31 01:19:35.196017 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:35.195911    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs\") pod \"network-metrics-daemon-vfk9h\" (UID: \"d9e6beef-017b-49a6-81a3-45b8faf145fc\") " pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:19:35.196017 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:35.195976    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret\") pod \"global-pull-secret-syncer-5crfw\" (UID: \"f8449422-f48d-4df2-8548-9de1c5339fd0\") " pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:35.196230 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:35.196084    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:19:35.196230 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:35.196123    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:19:35.196230 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:35.196161    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs podName:d9e6beef-017b-49a6-81a3-45b8faf145fc nodeName:}" failed. No retries permitted until 2026-10-31 01:19:36.196141634 +0000 UTC m=+4.053028791 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs") pod "network-metrics-daemon-vfk9h" (UID: "d9e6beef-017b-49a6-81a3-45b8faf145fc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:19:35.196230 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:35.196178    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret podName:f8449422-f48d-4df2-8548-9de1c5339fd0 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:36.196171384 +0000 UTC m=+4.053058519 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret") pod "global-pull-secret-syncer-5crfw" (UID: "f8449422-f48d-4df2-8548-9de1c5339fd0") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:19:35.297024 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:35.296980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xs45\" (UniqueName: \"kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45\") pod \"network-check-target-jn7dx\" (UID: \"2aa13b7e-bcfc-4ca6-96bd-086caf620c9c\") " pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:19:35.297210 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:35.297188    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:19:35.297294 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:35.297220    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:19:35.297294 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:35.297237    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9xs45 for pod openshift-network-diagnostics/network-check-target-jn7dx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:19:35.297393 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:35.297311    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45 podName:2aa13b7e-bcfc-4ca6-96bd-086caf620c9c nodeName:}" failed. No retries permitted until 2026-10-31 01:19:36.297288876 +0000 UTC m=+4.154176026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9xs45" (UniqueName: "kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45") pod "network-check-target-jn7dx" (UID: "2aa13b7e-bcfc-4ca6-96bd-086caf620c9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:19:35.617323 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:35.617220    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 01:14:34 +0000 UTC" deadline="2028-06-16 01:57:49.02959259 +0000 UTC"
Oct 31 01:19:35.617323 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:35.617263    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14256h38m13.412334519s"
Oct 31 01:19:35.729541 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:35.728906    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:19:35.729541 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:35.729034    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn7dx" podUID="2aa13b7e-bcfc-4ca6-96bd-086caf620c9c"
Oct 31 01:19:35.750589 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:35.750469    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fqpcx" event={"ID":"81c10625-0368-49c7-89cf-eb51f06005d0","Type":"ContainerStarted","Data":"5da57e3fdb12e9c5a9f6d66de998f92b009e424cd0f4f55a19e8644e439d3c93"}
Oct 31 01:19:35.763367 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:35.763206    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lhhnx" event={"ID":"db94c614-bfe5-4328-b1aa-770694e4fca1","Type":"ContainerStarted","Data":"801d38c689ae4e6ea6e4a69f864ddc10453c1012b70f6632a1dfe0a7496e7659"}
Oct 31 01:19:35.778280 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:35.778161    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv" event={"ID":"02c05e0c-b77a-45e4-b49b-1bed7e7588d0","Type":"ContainerStarted","Data":"0fd8f4b05e04837e0e0b26e5608e81495837e6095d72ec6fad2bef34dcaf7974"}
Oct 31 01:19:35.781313 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:35.781241    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xghmx" event={"ID":"86947040-78ac-43d5-b932-9acd63cf43e7","Type":"ContainerStarted","Data":"2a7902ff147c883af26f277241b0fa036f5e800913f8d7aeb1709878f1d15002"}
Oct 31 01:19:35.799537 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:35.799445    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ll9gg" event={"ID":"ec3e19fc-4890-424b-86ef-f5ad1e926158","Type":"ContainerStarted","Data":"f62e7b940902c55bf18ce7cbe90f35be40ea9216ce7ade5ee556fcac904d1e55"}
Oct 31 01:19:35.809334 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:35.809296    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmh79" event={"ID":"6b7841e1-63bb-4e71-972e-1020c47d2ccb","Type":"ContainerStarted","Data":"1bc364fd433ed3d2f63c9112e2d37ed896872d6d972a9b39d31c407825d5bd70"}
Oct 31 01:19:35.817713 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:35.817677    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tzz5q" event={"ID":"52e25fa9-6c04-453a-a0b2-cca32a428a5a","Type":"ContainerStarted","Data":"b5d3ecbcd60a975ff2c124eb66e9c9f22998bce80d5d69ad3359575bf6360e19"}
Oct 31 01:19:35.848325 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:35.848213    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wnmfp" event={"ID":"308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f","Type":"ContainerStarted","Data":"9849baee4ffe03763bf3fa9a1bc91f2b9d3a2c828d6cb0d18680c50d8505efa7"}
Oct 31 01:19:35.854997 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:35.854869    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m7xdt" event={"ID":"b0cbe408-97f0-48da-ab7d-ab4d144f0796","Type":"ContainerStarted","Data":"00643bc5ed3a191fc78de2b4eac3d3c4daab22f1bd6907b5e381adc3285f7d74"}
Oct 31 01:19:36.206934 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:36.206823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret\") pod \"global-pull-secret-syncer-5crfw\" (UID: \"f8449422-f48d-4df2-8548-9de1c5339fd0\") " pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:36.207143 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:36.207030    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs\") pod \"network-metrics-daemon-vfk9h\" (UID: \"d9e6beef-017b-49a6-81a3-45b8faf145fc\") " pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:19:36.207223 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:36.207169    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:19:36.207276 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:36.207240    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs podName:d9e6beef-017b-49a6-81a3-45b8faf145fc nodeName:}" failed. No retries permitted until 2026-10-31 01:19:38.207220078 +0000 UTC m=+6.064107219 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs") pod "network-metrics-daemon-vfk9h" (UID: "d9e6beef-017b-49a6-81a3-45b8faf145fc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:19:36.207750 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:36.207729    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:19:36.207870 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:36.207793    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret podName:f8449422-f48d-4df2-8548-9de1c5339fd0 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:38.20777685 +0000 UTC m=+6.064663985 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret") pod "global-pull-secret-syncer-5crfw" (UID: "f8449422-f48d-4df2-8548-9de1c5339fd0") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:19:36.308678 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:36.307866    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xs45\" (UniqueName: \"kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45\") pod \"network-check-target-jn7dx\" (UID: \"2aa13b7e-bcfc-4ca6-96bd-086caf620c9c\") " pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:19:36.308678 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:36.308058    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:19:36.308678 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:36.308083    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:19:36.308678 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:36.308097    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9xs45 for pod openshift-network-diagnostics/network-check-target-jn7dx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:19:36.308678 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:36.308196    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45 podName:2aa13b7e-bcfc-4ca6-96bd-086caf620c9c nodeName:}" failed. No retries permitted until 2026-10-31 01:19:38.308140264 +0000 UTC m=+6.165027430 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9xs45" (UniqueName: "kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45") pod "network-check-target-jn7dx" (UID: "2aa13b7e-bcfc-4ca6-96bd-086caf620c9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:19:36.617707 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:36.617614    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 01:14:34 +0000 UTC" deadline="2028-05-19 07:23:47.93514241 +0000 UTC"
Oct 31 01:19:36.617707 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:36.617674    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13590h4m11.317472711s"
Oct 31 01:19:36.736749 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:36.736716    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:19:36.736928 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:36.736866    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfk9h" podUID="d9e6beef-017b-49a6-81a3-45b8faf145fc"
Oct 31 01:19:36.737484 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:36.737265    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:36.737484 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:36.737372    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfw" podUID="f8449422-f48d-4df2-8548-9de1c5339fd0"
Oct 31 01:19:36.877806 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:36.877383    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal" event={"ID":"7b4242bdffafd37f633de48cdce1cc09","Type":"ContainerStarted","Data":"6eae2816c9f8ae6f26610a9d26d7a5ac924261135af2d8f4a979b68562100e8c"}
Oct 31 01:19:37.729466 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:37.729004    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:19:37.729466 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:37.729133    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn7dx" podUID="2aa13b7e-bcfc-4ca6-96bd-086caf620c9c"
Oct 31 01:19:37.890567 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:37.890438    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal" event={"ID":"a2a6228f7ebdca678429d632d3fb70c4","Type":"ContainerStarted","Data":"1df7e0014ee5a640c7d1d3465f00a2f672d0cf2e801516c8ee9152a472a69da8"}
Oct 31 01:19:37.909060 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:37.908992    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal" podStartSLOduration=3.9089690409999998 podStartE2EDuration="3.908969041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:36.895819092 +0000 UTC m=+4.752706249" watchObservedRunningTime="2026-10-31 01:19:37.908969041 +0000 UTC m=+5.765856197"
Oct 31 01:19:38.230569 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:38.229860    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs\") pod \"network-metrics-daemon-vfk9h\" (UID: \"d9e6beef-017b-49a6-81a3-45b8faf145fc\") " pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:19:38.230569 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:38.229901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret\") pod \"global-pull-secret-syncer-5crfw\" (UID: \"f8449422-f48d-4df2-8548-9de1c5339fd0\") " pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:38.230569 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:38.230003    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:19:38.230569 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:38.230052    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret podName:f8449422-f48d-4df2-8548-9de1c5339fd0 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:42.230040286 +0000 UTC m=+10.086927418 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret") pod "global-pull-secret-syncer-5crfw" (UID: "f8449422-f48d-4df2-8548-9de1c5339fd0") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:19:38.230569 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:38.230116    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:19:38.230569 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:38.230162    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs podName:d9e6beef-017b-49a6-81a3-45b8faf145fc nodeName:}" failed. No retries permitted until 2026-10-31 01:19:42.230153185 +0000 UTC m=+10.087040316 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs") pod "network-metrics-daemon-vfk9h" (UID: "d9e6beef-017b-49a6-81a3-45b8faf145fc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:19:38.330598 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:38.330555    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xs45\" (UniqueName: \"kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45\") pod \"network-check-target-jn7dx\" (UID: \"2aa13b7e-bcfc-4ca6-96bd-086caf620c9c\") " pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:19:38.330918 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:38.330735    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:19:38.330918 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:38.330758    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:19:38.330918 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:38.330771    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9xs45 for pod openshift-network-diagnostics/network-check-target-jn7dx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:19:38.330918 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:38.330836    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45 podName:2aa13b7e-bcfc-4ca6-96bd-086caf620c9c nodeName:}" failed. No retries permitted until 2026-10-31 01:19:42.330815795 +0000 UTC m=+10.187702934 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9xs45" (UniqueName: "kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45") pod "network-check-target-jn7dx" (UID: "2aa13b7e-bcfc-4ca6-96bd-086caf620c9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:19:38.730599 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:38.730569    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:19:38.731037 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:38.730722    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfk9h" podUID="d9e6beef-017b-49a6-81a3-45b8faf145fc"
Oct 31 01:19:38.734480 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:38.734299    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:38.734480 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:38.734431    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfw" podUID="f8449422-f48d-4df2-8548-9de1c5339fd0"
Oct 31 01:19:38.897597 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:38.896927    2578 generic.go:356] "Generic (PLEG): container finished" podID="ec3e19fc-4890-424b-86ef-f5ad1e926158" containerID="ab40995ed15e2a02727e79ce17c0bccd74406086035b7c4a11d7a04bbbcc8561" exitCode=0
Oct 31 01:19:38.897597 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:38.897033    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ll9gg" event={"ID":"ec3e19fc-4890-424b-86ef-f5ad1e926158","Type":"ContainerDied","Data":"ab40995ed15e2a02727e79ce17c0bccd74406086035b7c4a11d7a04bbbcc8561"}
Oct 31 01:19:38.900795 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:38.900694    2578 generic.go:356] "Generic (PLEG): container finished" podID="a2a6228f7ebdca678429d632d3fb70c4" containerID="1df7e0014ee5a640c7d1d3465f00a2f672d0cf2e801516c8ee9152a472a69da8" exitCode=0
Oct 31 01:19:38.900795 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:38.900739    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal" event={"ID":"a2a6228f7ebdca678429d632d3fb70c4","Type":"ContainerDied","Data":"1df7e0014ee5a640c7d1d3465f00a2f672d0cf2e801516c8ee9152a472a69da8"}
Oct 31 01:19:39.728926 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:39.728893    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:19:39.729122 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:39.729001    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn7dx" podUID="2aa13b7e-bcfc-4ca6-96bd-086caf620c9c"
Oct 31 01:19:39.909794 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:39.909758    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wnmfp" event={"ID":"308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f","Type":"ContainerStarted","Data":"f1ffbbea5240ffa9883aa10a5b7cccc5bf89d6249fad96cf99bf38e2d345e509"}
Oct 31 01:19:39.915909 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:39.915877    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal" event={"ID":"a2a6228f7ebdca678429d632d3fb70c4","Type":"ContainerStarted","Data":"6fb46a21ef135cc64da83a5afcb4a0a0259645f1684dd7e48140dda332177113"}
Oct 31 01:19:39.930965 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:39.930909    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wnmfp" podStartSLOduration=2.682358741 podStartE2EDuration="6.930891754s" totalImagesPullingTime="4.248533013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:39.930600069 +0000 UTC m=+7.787487222" watchObservedRunningTime="2026-10-31 01:19:39.930891754 +0000 UTC m=+7.787778910"
Oct 31 01:19:40.730778 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:40.730743    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:40.730955 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:40.730876    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfw" podUID="f8449422-f48d-4df2-8548-9de1c5339fd0"
Oct 31 01:19:40.731415 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:40.731394    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:19:40.731608 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:40.731585    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfk9h" podUID="d9e6beef-017b-49a6-81a3-45b8faf145fc"
Oct 31 01:19:40.921808 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:40.921723    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tzz5q" event={"ID":"52e25fa9-6c04-453a-a0b2-cca32a428a5a","Type":"ContainerStarted","Data":"36e977dce2b5f27d5052ea5ee42749722fd95d522b335397466012d3800fd86c"}
Oct 31 01:19:40.938126 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:40.937154    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal" podStartSLOduration=6.937134997 podStartE2EDuration="6.937134997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:39.947584736 +0000 UTC m=+7.804471934" watchObservedRunningTime="2026-10-31 01:19:40.937134997 +0000 UTC m=+8.794022152"
Oct 31 01:19:40.938126 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:40.937386    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tzz5q" podStartSLOduration=2.182524667 podStartE2EDuration="7.937380258s" totalImagesPullingTime="5.754855591s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:40.936762521 +0000 UTC m=+8.793649674" watchObservedRunningTime="2026-10-31 01:19:40.937380258 +0000 UTC m=+8.794267411"
Oct 31 01:19:41.728963 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:41.728932    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:19:41.729152 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:41.729058    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn7dx" podUID="2aa13b7e-bcfc-4ca6-96bd-086caf620c9c"
Oct 31 01:19:41.926881 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:41.926842    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv" event={"ID":"02c05e0c-b77a-45e4-b49b-1bed7e7588d0","Type":"ContainerStarted","Data":"d7bed6e85ab48833041bd668c6780ad3e4d79db08f3296f28b4749559143dcf3"}
Oct 31 01:19:42.021532 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:42.021432    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wnmfp"
Oct 31 01:19:42.186047 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:42.186002    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wnmfp"
Oct 31 01:19:42.186763 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:42.186744    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wnmfp"
Oct 31 01:19:42.265807 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:42.265760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs\") pod \"network-metrics-daemon-vfk9h\" (UID: \"d9e6beef-017b-49a6-81a3-45b8faf145fc\") " pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:19:42.265993 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:42.265828    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret\") pod \"global-pull-secret-syncer-5crfw\" (UID: \"f8449422-f48d-4df2-8548-9de1c5339fd0\") " pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:42.266052 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:42.265995    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:19:42.266096 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:42.266057    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret podName:f8449422-f48d-4df2-8548-9de1c5339fd0 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:50.26604005 +0000 UTC m=+18.122927197 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret") pod "global-pull-secret-syncer-5crfw" (UID: "f8449422-f48d-4df2-8548-9de1c5339fd0") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:19:42.266152 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:42.266119    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:19:42.266152 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:42.266151    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs podName:d9e6beef-017b-49a6-81a3-45b8faf145fc nodeName:}" failed. No retries permitted until 2026-10-31 01:19:50.266139597 +0000 UTC m=+18.123026731 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs") pod "network-metrics-daemon-vfk9h" (UID: "d9e6beef-017b-49a6-81a3-45b8faf145fc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:19:42.366663 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:42.366578    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xs45\" (UniqueName: \"kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45\") pod \"network-check-target-jn7dx\" (UID: \"2aa13b7e-bcfc-4ca6-96bd-086caf620c9c\") " pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:19:42.366818 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:42.366795    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:19:42.366818 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:42.366814    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:19:42.366937 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:42.366827    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9xs45 for pod openshift-network-diagnostics/network-check-target-jn7dx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:19:42.366937 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:42.366885    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45 podName:2aa13b7e-bcfc-4ca6-96bd-086caf620c9c nodeName:}" failed. No retries permitted until 2026-10-31 01:19:50.366868339 +0000 UTC m=+18.223755472 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9xs45" (UniqueName: "kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45") pod "network-check-target-jn7dx" (UID: "2aa13b7e-bcfc-4ca6-96bd-086caf620c9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:19:42.732951 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:42.732877    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:19:42.733107 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:42.733062    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfk9h" podUID="d9e6beef-017b-49a6-81a3-45b8faf145fc"
Oct 31 01:19:42.733561 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:42.733410    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:42.733561 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:42.733524    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfw" podUID="f8449422-f48d-4df2-8548-9de1c5339fd0"
Oct 31 01:19:42.930778 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:42.930747    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wnmfp"
Oct 31 01:19:43.153118 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:43.153057    2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 01:19:43.652842 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:43.652583    2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T01:19:43.153077031Z","UUID":"f0d48dd9-4d43-4e3f-a3ac-1ca762e34bde","Handler":null,"Name":"","Endpoint":""}
Oct 31 01:19:43.654649 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:43.654243    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 01:19:43.654649 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:43.654271    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 01:19:43.729695 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:43.729287    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:19:43.729695 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:43.729383    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn7dx" podUID="2aa13b7e-bcfc-4ca6-96bd-086caf620c9c"
Oct 31 01:19:43.933913 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:43.933826    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fqpcx" event={"ID":"81c10625-0368-49c7-89cf-eb51f06005d0","Type":"ContainerStarted","Data":"fca9889ecd5b0b71ccacc3d0fed8c34edb6ec825e1d4ded8ca2317ff5a88fa01"}
Oct 31 01:19:43.946182 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:43.946140    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv" event={"ID":"02c05e0c-b77a-45e4-b49b-1bed7e7588d0","Type":"ContainerStarted","Data":"016c2752f3742ce6211991f4f8a50e10b24000ccda3ce384c13b80acc2eab129"}
Oct 31 01:19:43.948905 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:43.948866    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xghmx" event={"ID":"86947040-78ac-43d5-b932-9acd63cf43e7","Type":"ContainerStarted","Data":"0b50572448e894a2c1c62b561aa5a5c7e67eb9bdfbe2874bc53f2ac2023eee1c"}
Oct 31 01:19:43.951359 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:43.951330    2578 generic.go:356] "Generic (PLEG): container finished" podID="ec3e19fc-4890-424b-86ef-f5ad1e926158" containerID="ef579280d5803ddba0a67dc0e4fe5ae9e485903371cc26be617f979a841bd747" exitCode=0
Oct 31 01:19:43.951482 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:43.951432    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ll9gg" event={"ID":"ec3e19fc-4890-424b-86ef-f5ad1e926158","Type":"ContainerDied","Data":"ef579280d5803ddba0a67dc0e4fe5ae9e485903371cc26be617f979a841bd747"}
Oct 31 01:19:43.961838 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:43.961779    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-fqpcx" podStartSLOduration=2.90640778 podStartE2EDuration="10.961761011s" totalImagesPullingTime="8.055353231s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:43.961462262 +0000 UTC m=+11.818349414" watchObservedRunningTime="2026-10-31 01:19:43.961761011 +0000 UTC m=+11.818648165"
Oct 31 01:19:43.977784 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:43.977729    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-xghmx" podStartSLOduration=2.292669922 podStartE2EDuration="10.977714999s" totalImagesPullingTime="8.685045077s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:43.977409796 +0000 UTC m=+11.834296950" watchObservedRunningTime="2026-10-31 01:19:43.977714999 +0000 UTC m=+11.834602150"
Oct 31 01:19:44.728705 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:44.728676    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:44.728824 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:44.728804    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfw" podUID="f8449422-f48d-4df2-8548-9de1c5339fd0"
Oct 31 01:19:44.729564 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:44.729410    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:19:44.729564 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:44.729551    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfk9h" podUID="d9e6beef-017b-49a6-81a3-45b8faf145fc"
Oct 31 01:19:44.955497 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:44.955462    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv" event={"ID":"02c05e0c-b77a-45e4-b49b-1bed7e7588d0","Type":"ContainerStarted","Data":"f787f8db669e80e3349f6e91047293297f2be085cd46da86d93209ec49ed428c"}
Oct 31 01:19:44.957566 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:44.957537    2578 generic.go:356] "Generic (PLEG): container finished" podID="ec3e19fc-4890-424b-86ef-f5ad1e926158" containerID="5506323c3914d48883f1e18256a04bf102a41a698f26fc706a8e4b288faab1eb" exitCode=0
Oct 31 01:19:44.957740 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:44.957643    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ll9gg" event={"ID":"ec3e19fc-4890-424b-86ef-f5ad1e926158","Type":"ContainerDied","Data":"5506323c3914d48883f1e18256a04bf102a41a698f26fc706a8e4b288faab1eb"}
Oct 31 01:19:44.975003 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:44.974950    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv" podStartSLOduration=3.030749186 podStartE2EDuration="11.974930674s" totalImagesPullingTime="8.944181488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:44.974695386 +0000 UTC m=+12.831582538" watchObservedRunningTime="2026-10-31 01:19:44.974930674 +0000 UTC m=+12.831817827"
Oct 31 01:19:45.729625 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:45.729588    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:19:45.729800 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:45.729721    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn7dx" podUID="2aa13b7e-bcfc-4ca6-96bd-086caf620c9c"
Oct 31 01:19:45.962872 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:45.962835    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ll9gg" event={"ID":"ec3e19fc-4890-424b-86ef-f5ad1e926158","Type":"ContainerStarted","Data":"a034c68cb6dc0afb42e3348e3ad8c60819824cf0eb95a3b1114fdc1fcc8c247a"}
Oct 31 01:19:45.964864 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:45.964833    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m7xdt" event={"ID":"b0cbe408-97f0-48da-ab7d-ab4d144f0796","Type":"ContainerStarted","Data":"8c603c2d1efca6530041998e403f4ecb45bc7d2950415d56a3db16d5bd2df166"}
Oct 31 01:19:45.967386 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:45.967361    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lhhnx" event={"ID":"db94c614-bfe5-4328-b1aa-770694e4fca1","Type":"ContainerStarted","Data":"cf5bd081f36a1d71757eefe646674b31bef042feafe1acfad9393e5f8ff7f6b1"}
Oct 31 01:19:46.001179 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:46.001106    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lhhnx" podStartSLOduration=4.389170353 podStartE2EDuration="13.001080887s" totalImagesPullingTime="8.611910534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:46.000570646 +0000 UTC m=+13.857457799" watchObservedRunningTime="2026-10-31 01:19:46.001080887 +0000 UTC m=+13.857968039"
Oct 31 01:19:46.018402 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:46.018350    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-m7xdt" podStartSLOduration=2.339403694 podStartE2EDuration="13.018335093s" totalImagesPullingTime="10.678931399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:46.017805302 +0000 UTC m=+13.874692456" watchObservedRunningTime="2026-10-31 01:19:46.018335093 +0000 UTC m=+13.875222245"
Oct 31 01:19:46.729521 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:46.729486    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:19:46.729521 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:46.729509    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:46.729724 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:46.729636    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfk9h" podUID="d9e6beef-017b-49a6-81a3-45b8faf145fc"
Oct 31 01:19:46.729776 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:46.729754    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfw" podUID="f8449422-f48d-4df2-8548-9de1c5339fd0"
Oct 31 01:19:46.972669 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:46.972633    2578 generic.go:356] "Generic (PLEG): container finished" podID="ec3e19fc-4890-424b-86ef-f5ad1e926158" containerID="a034c68cb6dc0afb42e3348e3ad8c60819824cf0eb95a3b1114fdc1fcc8c247a" exitCode=0
Oct 31 01:19:46.973323 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:46.972735    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ll9gg" event={"ID":"ec3e19fc-4890-424b-86ef-f5ad1e926158","Type":"ContainerDied","Data":"a034c68cb6dc0afb42e3348e3ad8c60819824cf0eb95a3b1114fdc1fcc8c247a"}
Oct 31 01:19:47.729249 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:47.729028    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:19:47.729467 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:47.729332    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn7dx" podUID="2aa13b7e-bcfc-4ca6-96bd-086caf620c9c"
Oct 31 01:19:48.729077 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:48.729044    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:19:48.729614 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:48.729193    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfk9h" podUID="d9e6beef-017b-49a6-81a3-45b8faf145fc"
Oct 31 01:19:48.729614 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:48.729507    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:48.729614 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:48.729579    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfw" podUID="f8449422-f48d-4df2-8548-9de1c5339fd0"
Oct 31 01:19:49.729499 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:49.729459    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:19:49.729943 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:49.729579    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn7dx" podUID="2aa13b7e-bcfc-4ca6-96bd-086caf620c9c"
Oct 31 01:19:49.982566 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:49.982470    2578 generic.go:356] "Generic (PLEG): container finished" podID="ec3e19fc-4890-424b-86ef-f5ad1e926158" containerID="25c8e34977842b7c816a16c4e4032a4c099da48e0ed22fe680b12ded1b4ff90c" exitCode=0
Oct 31 01:19:49.982719 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:49.982552    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ll9gg" event={"ID":"ec3e19fc-4890-424b-86ef-f5ad1e926158","Type":"ContainerDied","Data":"25c8e34977842b7c816a16c4e4032a4c099da48e0ed22fe680b12ded1b4ff90c"}
Oct 31 01:19:50.332080 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:50.332052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs\") pod \"network-metrics-daemon-vfk9h\" (UID: \"d9e6beef-017b-49a6-81a3-45b8faf145fc\") " pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:19:50.332229 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:50.332087    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret\") pod \"global-pull-secret-syncer-5crfw\" (UID: \"f8449422-f48d-4df2-8548-9de1c5339fd0\") " pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:50.332229 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:50.332184    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:19:50.332229 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:50.332187    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:19:50.332348 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:50.332235    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret podName:f8449422-f48d-4df2-8548-9de1c5339fd0 nodeName:}" failed. No retries permitted until 2026-10-31 01:20:06.332222832 +0000 UTC m=+34.189109963 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret") pod "global-pull-secret-syncer-5crfw" (UID: "f8449422-f48d-4df2-8548-9de1c5339fd0") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:19:50.332348 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:50.332249    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs podName:d9e6beef-017b-49a6-81a3-45b8faf145fc nodeName:}" failed. No retries permitted until 2026-10-31 01:20:06.332242179 +0000 UTC m=+34.189129309 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs") pod "network-metrics-daemon-vfk9h" (UID: "d9e6beef-017b-49a6-81a3-45b8faf145fc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:19:50.432807 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:50.432771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xs45\" (UniqueName: \"kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45\") pod \"network-check-target-jn7dx\" (UID: \"2aa13b7e-bcfc-4ca6-96bd-086caf620c9c\") " pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:19:50.432952 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:50.432916    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:19:50.432952 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:50.432931    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:19:50.432952 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:50.432944    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9xs45 for pod openshift-network-diagnostics/network-check-target-jn7dx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:19:50.433038 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:50.432990    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45 podName:2aa13b7e-bcfc-4ca6-96bd-086caf620c9c nodeName:}" failed. No retries permitted until 2026-10-31 01:20:06.432976913 +0000 UTC m=+34.289864046 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9xs45" (UniqueName: "kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45") pod "network-check-target-jn7dx" (UID: "2aa13b7e-bcfc-4ca6-96bd-086caf620c9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:19:50.728665 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:50.728635    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:19:50.728816 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:50.728775    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfk9h" podUID="d9e6beef-017b-49a6-81a3-45b8faf145fc"
Oct 31 01:19:50.728866 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:50.728817    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:50.728927 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:50.728908    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfw" podUID="f8449422-f48d-4df2-8548-9de1c5339fd0"
Oct 31 01:19:50.990918 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:50.990887    2578 generic.go:356] "Generic (PLEG): container finished" podID="ec3e19fc-4890-424b-86ef-f5ad1e926158" containerID="c29de0b235c3d20d87f4a0cec521f7fb1a0524eca5b5d527fefc142ec4e6d175" exitCode=0
Oct 31 01:19:50.991309 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:50.990933    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ll9gg" event={"ID":"ec3e19fc-4890-424b-86ef-f5ad1e926158","Type":"ContainerDied","Data":"c29de0b235c3d20d87f4a0cec521f7fb1a0524eca5b5d527fefc142ec4e6d175"}
Oct 31 01:19:51.728660 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:51.728621    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:19:51.728850 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:51.728730    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn7dx" podUID="2aa13b7e-bcfc-4ca6-96bd-086caf620c9c"
Oct 31 01:19:51.996256 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:51.996164    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmh79" event={"ID":"6b7841e1-63bb-4e71-972e-1020c47d2ccb","Type":"ContainerStarted","Data":"d54dd11ecd0eb4de73bf305d4ca107ee4494980f5a8b462ae37d1545c1755247"}
Oct 31 01:19:51.996256 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:51.996249    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmh79" event={"ID":"6b7841e1-63bb-4e71-972e-1020c47d2ccb","Type":"ContainerStarted","Data":"9c2af1bcff73cfedca886aacf206fe97114bcf2fc258053055f10a6f23827bfe"}
Oct 31 01:19:51.996831 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:51.996291    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmh79" event={"ID":"6b7841e1-63bb-4e71-972e-1020c47d2ccb","Type":"ContainerStarted","Data":"19cc4eee8eff0e5f63a8ebfe528dce7770ad6833119303ff98321a7b67d5f91a"}
Oct 31 01:19:51.996831 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:51.996332    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmh79" event={"ID":"6b7841e1-63bb-4e71-972e-1020c47d2ccb","Type":"ContainerStarted","Data":"12b73d91427fcb17d6e56dfe665381472667f7f651cd34c353ebfe6bde0ccdae"}
Oct 31 01:19:51.996831 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:51.996379    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmh79" event={"ID":"6b7841e1-63bb-4e71-972e-1020c47d2ccb","Type":"ContainerStarted","Data":"99d40c2ebe4645a962cc74a22562df40f74ef12b1db9e16a3dd656e4948bf39d"}
Oct 31 01:19:51.996831 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:51.996430    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmh79" event={"ID":"6b7841e1-63bb-4e71-972e-1020c47d2ccb","Type":"ContainerStarted","Data":"5245e3592481ca7f67578badf3c6b9593482bd48cc77ae65ad219fa31c6b7e8e"}
Oct 31 01:19:51.999342 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:51.999318    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ll9gg" event={"ID":"ec3e19fc-4890-424b-86ef-f5ad1e926158","Type":"ContainerStarted","Data":"3510f85a3bdfaf1425655bd417923d0ec5c0ab85f73f475061b10e4e63542291"}
Oct 31 01:19:52.022419 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:52.022362    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ll9gg" podStartSLOduration=7.428985946 podStartE2EDuration="19.022348997s" totalImagesPullingTime="11.593363051s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:52.021849349 +0000 UTC m=+19.878736501" watchObservedRunningTime="2026-10-31 01:19:52.022348997 +0000 UTC m=+19.879236149"
Oct 31 01:19:52.730069 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:52.730031    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:19:52.730243 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:52.730135    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfk9h" podUID="d9e6beef-017b-49a6-81a3-45b8faf145fc"
Oct 31 01:19:52.730243 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:52.730165    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:52.730243 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:52.730220    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfw" podUID="f8449422-f48d-4df2-8548-9de1c5339fd0"
Oct 31 01:19:53.729281 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:53.729249    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:19:53.729774 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:53.729355    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn7dx" podUID="2aa13b7e-bcfc-4ca6-96bd-086caf620c9c"
Oct 31 01:19:54.006716 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:54.006628    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmh79" event={"ID":"6b7841e1-63bb-4e71-972e-1020c47d2ccb","Type":"ContainerStarted","Data":"9c2904bd4e96ff708929efdd883f14654e1b09f690e940fef4288ed8ab5ed0ff"}
Oct 31 01:19:54.729713 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:54.729675    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:19:54.730083 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:54.729792    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfk9h" podUID="d9e6beef-017b-49a6-81a3-45b8faf145fc"
Oct 31 01:19:54.730083 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:54.729845    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:54.730083 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:54.729931    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfw" podUID="f8449422-f48d-4df2-8548-9de1c5339fd0"
Oct 31 01:19:55.728918 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:55.728877    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:19:55.729093 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:55.728985    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn7dx" podUID="2aa13b7e-bcfc-4ca6-96bd-086caf620c9c"
Oct 31 01:19:56.014304 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:56.014206    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tmh79" event={"ID":"6b7841e1-63bb-4e71-972e-1020c47d2ccb","Type":"ContainerStarted","Data":"256a86e7055a37ebd3efcfd5d5fc257952f182cb60a59af99a709b2728bee030"}
Oct 31 01:19:56.015075 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:56.014478    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:56.029852 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:56.029828    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:56.047815 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:56.047768    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tmh79" podStartSLOduration=7.047053694 podStartE2EDuration="23.047756202s" totalImagesPullingTime="16.000702508s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:56.046714886 +0000 UTC m=+23.903602037" watchObservedRunningTime="2026-10-31 01:19:56.047756202 +0000 UTC m=+23.904643353"
Oct 31 01:19:56.730696 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:56.729982    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:19:56.730696 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:56.730139    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfk9h" podUID="d9e6beef-017b-49a6-81a3-45b8faf145fc"
Oct 31 01:19:56.730696 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:56.730562    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:56.730696 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:56.730657    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfw" podUID="f8449422-f48d-4df2-8548-9de1c5339fd0"
Oct 31 01:19:57.016807 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:57.016707    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:57.016807 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:57.016788    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:57.032576 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:57.032550    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:19:57.201543 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:57.201509    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vfk9h"]
Oct 31 01:19:57.201693 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:57.201638    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:19:57.201788 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:57.201753    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfk9h" podUID="d9e6beef-017b-49a6-81a3-45b8faf145fc"
Oct 31 01:19:57.205638 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:57.205612    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5crfw"]
Oct 31 01:19:57.205779 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:57.205665    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jn7dx"]
Oct 31 01:19:57.205779 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:57.205718    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:19:57.205865 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:57.205796    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn7dx" podUID="2aa13b7e-bcfc-4ca6-96bd-086caf620c9c"
Oct 31 01:19:57.205865 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:57.205845    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:57.205939 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:57.205905    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfw" podUID="f8449422-f48d-4df2-8548-9de1c5339fd0"
Oct 31 01:19:58.729697 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:58.729660    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:19:58.729697 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:58.729690    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:19:58.730110 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:19:58.729675    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:19:58.730110 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:58.729791    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn7dx" podUID="2aa13b7e-bcfc-4ca6-96bd-086caf620c9c"
Oct 31 01:19:58.730110 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:58.729861    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfk9h" podUID="d9e6beef-017b-49a6-81a3-45b8faf145fc"
Oct 31 01:19:58.730110 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:19:58.729915    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfw" podUID="f8449422-f48d-4df2-8548-9de1c5339fd0"
Oct 31 01:20:00.728814 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:00.728777    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:20:00.728814 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:00.728796    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:20:00.728814 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:00.728784    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:20:00.729392 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:00.728896    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfw" podUID="f8449422-f48d-4df2-8548-9de1c5339fd0"
Oct 31 01:20:00.729392 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:00.728935    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn7dx" podUID="2aa13b7e-bcfc-4ca6-96bd-086caf620c9c"
Oct 31 01:20:00.729392 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:00.729037    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfk9h" podUID="d9e6beef-017b-49a6-81a3-45b8faf145fc"
Oct 31 01:20:01.169112 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:01.169084    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xghmx_86947040-78ac-43d5-b932-9acd63cf43e7/dns-node-resolver/0.log"
Oct 31 01:20:02.132813 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:02.132785    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tzz5q_52e25fa9-6c04-453a-a0b2-cca32a428a5a/node-ca/0.log"
Oct 31 01:20:02.729873 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:02.729834    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:20:02.730047 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:02.729969    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:20:02.730047 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:02.729980    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfw" podUID="f8449422-f48d-4df2-8548-9de1c5339fd0"
Oct 31 01:20:02.730131 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:02.730069    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfk9h" podUID="d9e6beef-017b-49a6-81a3-45b8faf145fc"
Oct 31 01:20:02.730131 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:02.730095    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:20:02.730190 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:02.730162    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn7dx" podUID="2aa13b7e-bcfc-4ca6-96bd-086caf620c9c"
Oct 31 01:20:04.728692 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:04.728661    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:20:04.728692 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:04.728679    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:20:04.729281 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:04.728661    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:20:04.729281 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:04.728789    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfk9h" podUID="d9e6beef-017b-49a6-81a3-45b8faf145fc"
Oct 31 01:20:04.729281 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:04.728901    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn7dx" podUID="2aa13b7e-bcfc-4ca6-96bd-086caf620c9c"
Oct 31 01:20:04.729281 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:04.728975    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfw" podUID="f8449422-f48d-4df2-8548-9de1c5339fd0"
Oct 31 01:20:06.347007 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:06.346966    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs\") pod \"network-metrics-daemon-vfk9h\" (UID: \"d9e6beef-017b-49a6-81a3-45b8faf145fc\") " pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:20:06.347007 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:06.347010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret\") pod \"global-pull-secret-syncer-5crfw\" (UID: \"f8449422-f48d-4df2-8548-9de1c5339fd0\") " pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:20:06.347424 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:06.347102    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:20:06.347424 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:06.347107    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:20:06.347424 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:06.347152    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret podName:f8449422-f48d-4df2-8548-9de1c5339fd0 nodeName:}" failed. No retries permitted until 2026-10-31 01:20:38.347139377 +0000 UTC m=+66.204026507 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret") pod "global-pull-secret-syncer-5crfw" (UID: "f8449422-f48d-4df2-8548-9de1c5339fd0") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:20:06.347424 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:06.347166    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs podName:d9e6beef-017b-49a6-81a3-45b8faf145fc nodeName:}" failed. No retries permitted until 2026-10-31 01:20:38.347159941 +0000 UTC m=+66.204047071 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs") pod "network-metrics-daemon-vfk9h" (UID: "d9e6beef-017b-49a6-81a3-45b8faf145fc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:20:06.447934 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:06.447893    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xs45\" (UniqueName: \"kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45\") pod \"network-check-target-jn7dx\" (UID: \"2aa13b7e-bcfc-4ca6-96bd-086caf620c9c\") " pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:20:06.448094 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:06.448063    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:20:06.448094 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:06.448085    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:20:06.448094 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:06.448095    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9xs45 for pod openshift-network-diagnostics/network-check-target-jn7dx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:20:06.448192 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:06.448144    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45 podName:2aa13b7e-bcfc-4ca6-96bd-086caf620c9c nodeName:}" failed. No retries permitted until 2026-10-31 01:20:38.448131892 +0000 UTC m=+66.305019025 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9xs45" (UniqueName: "kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45") pod "network-check-target-jn7dx" (UID: "2aa13b7e-bcfc-4ca6-96bd-086caf620c9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:20:06.729743 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:06.729708    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:20:06.729743 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:06.729730    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:20:06.729951 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:06.729793    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:20:06.729951 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:06.729891    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfk9h" podUID="d9e6beef-017b-49a6-81a3-45b8faf145fc"
Oct 31 01:20:06.730030 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:06.730006    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfw" podUID="f8449422-f48d-4df2-8548-9de1c5339fd0"
Oct 31 01:20:06.730104 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:06.730088    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn7dx" podUID="2aa13b7e-bcfc-4ca6-96bd-086caf620c9c"
Oct 31 01:20:08.728789 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:08.728752    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:20:08.729290 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:08.728752    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:20:08.729290 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:08.728862    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfk9h" podUID="d9e6beef-017b-49a6-81a3-45b8faf145fc"
Oct 31 01:20:08.729290 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:08.728923    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfw" podUID="f8449422-f48d-4df2-8548-9de1c5339fd0"
Oct 31 01:20:08.729290 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:08.728752    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:20:08.729290 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:08.728977    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn7dx" podUID="2aa13b7e-bcfc-4ca6-96bd-086caf620c9c"
Oct 31 01:20:10.729196 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:10.729161    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:20:10.729196 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:10.729188    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:20:10.729695 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:10.729257    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfw" podUID="f8449422-f48d-4df2-8548-9de1c5339fd0"
Oct 31 01:20:10.729695 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:10.729312    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:20:10.729695 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:10.729420    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfk9h" podUID="d9e6beef-017b-49a6-81a3-45b8faf145fc"
Oct 31 01:20:10.729695 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:10.729501    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn7dx" podUID="2aa13b7e-bcfc-4ca6-96bd-086caf620c9c"
Oct 31 01:20:12.730929 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:12.730891    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:20:12.731331 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:12.730975    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:20:12.731331 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:12.731015    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:20:12.731331 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:12.731036    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn7dx" podUID="2aa13b7e-bcfc-4ca6-96bd-086caf620c9c"
Oct 31 01:20:12.731331 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:12.731092    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfw" podUID="f8449422-f48d-4df2-8548-9de1c5339fd0"
Oct 31 01:20:12.731331 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:12.731181    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfk9h" podUID="d9e6beef-017b-49a6-81a3-45b8faf145fc"
Oct 31 01:20:14.729331 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:14.729289    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:20:14.729331 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:14.729312    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:20:14.729774 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:14.729396    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:20:14.729774 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:14.729401    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfk9h" podUID="d9e6beef-017b-49a6-81a3-45b8faf145fc"
Oct 31 01:20:14.729774 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:14.729483    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn7dx" podUID="2aa13b7e-bcfc-4ca6-96bd-086caf620c9c"
Oct 31 01:20:14.729774 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:14.729541    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfw" podUID="f8449422-f48d-4df2-8548-9de1c5339fd0"
Oct 31 01:20:16.729141 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:16.729103    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:20:16.729141 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:16.729107    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:20:16.729603 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:16.729103    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:20:16.729603 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:16.729243    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5crfw" podUID="f8449422-f48d-4df2-8548-9de1c5339fd0"
Oct 31 01:20:16.729603 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:16.729259    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jn7dx" podUID="2aa13b7e-bcfc-4ca6-96bd-086caf620c9c"
Oct 31 01:20:16.729603 ip-10-0-133-98 kubenswrapper[2578]: E1031 01:20:16.729358    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfk9h" podUID="d9e6beef-017b-49a6-81a3-45b8faf145fc"
Oct 31 01:20:17.933174 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:17.933143    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-98.us-east-2.compute.internal" event="NodeReady"
Oct 31 01:20:17.933705 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:17.933295    2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 01:20:17.985035 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:17.985000    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8n2r2"]
Oct 31 01:20:17.986951 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:17.986935    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:17.988741 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:17.988715    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-v9wzp"]
Oct 31 01:20:17.990339 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:17.990320    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v9wzp"
Oct 31 01:20:17.995882 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:17.995861    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 01:20:17.995969 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:17.995895    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 01:20:17.996020 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:17.995971    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 01:20:17.996126 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:17.996109    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bztmp\""
Oct 31 01:20:17.996197 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:17.996146    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 01:20:17.999735 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:17.999707    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8n2r2"]
Oct 31 01:20:18.005676 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.005650    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v9wzp"]
Oct 31 01:20:18.006255 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.006238    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kmxk8"]
Oct 31 01:20:18.008076 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.008059    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kmxk8"
Oct 31 01:20:18.018804 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.018781    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kmxk8"]
Oct 31 01:20:18.025617 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.025593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:18.025739 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.025642    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c8rn\" (UniqueName: \"kubernetes.io/projected/60e7e4bc-764b-49e0-bb92-dea358fafcc5-kube-api-access-9c8rn\") pod \"ingress-canary-v9wzp\" (UID: \"60e7e4bc-764b-49e0-bb92-dea358fafcc5\") " pod="openshift-ingress-canary/ingress-canary-v9wzp"
Oct 31 01:20:18.025739 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.025720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-crio-socket\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:18.025820 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.025748    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:18.025820 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.025779    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g8zb\" (UniqueName: \"kubernetes.io/projected/aa56c5f6-4784-4b24-9daa-d17f7b296d8b-kube-api-access-7g8zb\") pod \"dns-default-kmxk8\" (UID: \"aa56c5f6-4784-4b24-9daa-d17f7b296d8b\") " pod="openshift-dns/dns-default-kmxk8"
Oct 31 01:20:18.025894 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.025865    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aa56c5f6-4784-4b24-9daa-d17f7b296d8b-config-volume\") pod \"dns-default-kmxk8\" (UID: \"aa56c5f6-4784-4b24-9daa-d17f7b296d8b\") " pod="openshift-dns/dns-default-kmxk8"
Oct 31 01:20:18.025933 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.025899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa56c5f6-4784-4b24-9daa-d17f7b296d8b-tmp-dir\") pod \"dns-default-kmxk8\" (UID: \"aa56c5f6-4784-4b24-9daa-d17f7b296d8b\") " pod="openshift-dns/dns-default-kmxk8"
Oct 31 01:20:18.025933 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.025922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pdph\" (UniqueName: \"kubernetes.io/projected/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-kube-api-access-7pdph\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:18.026001 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.025971    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-data-volume\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:18.026045 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.026023    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60e7e4bc-764b-49e0-bb92-dea358fafcc5-cert\") pod \"ingress-canary-v9wzp\" (UID: \"60e7e4bc-764b-49e0-bb92-dea358fafcc5\") " pod="openshift-ingress-canary/ingress-canary-v9wzp"
Oct 31 01:20:18.026045 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.026042    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa56c5f6-4784-4b24-9daa-d17f7b296d8b-metrics-tls\") pod \"dns-default-kmxk8\" (UID: \"aa56c5f6-4784-4b24-9daa-d17f7b296d8b\") " pod="openshift-dns/dns-default-kmxk8"
Oct 31 01:20:18.095975 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.095936    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 01:20:18.095975 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.095936    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 01:20:18.096184 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.096074    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 01:20:18.096184 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.096088    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 01:20:18.096184 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.096100    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 01:20:18.096184 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.096113    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-79nk6\""
Oct 31 01:20:18.096184 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.096152    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-whdn9\""
Oct 31 01:20:18.126526 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.126493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c8rn\" (UniqueName: \"kubernetes.io/projected/60e7e4bc-764b-49e0-bb92-dea358fafcc5-kube-api-access-9c8rn\") pod \"ingress-canary-v9wzp\" (UID: \"60e7e4bc-764b-49e0-bb92-dea358fafcc5\") " pod="openshift-ingress-canary/ingress-canary-v9wzp"
Oct 31 01:20:18.126599 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.126531    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-crio-socket\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:18.126713 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.126695    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:18.126761 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.126745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g8zb\" (UniqueName: \"kubernetes.io/projected/aa56c5f6-4784-4b24-9daa-d17f7b296d8b-kube-api-access-7g8zb\") pod \"dns-default-kmxk8\" (UID: \"aa56c5f6-4784-4b24-9daa-d17f7b296d8b\") " pod="openshift-dns/dns-default-kmxk8"
Oct 31 01:20:18.126794 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.126701    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-crio-socket\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:18.126887 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.126872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aa56c5f6-4784-4b24-9daa-d17f7b296d8b-config-volume\") pod \"dns-default-kmxk8\" (UID: \"aa56c5f6-4784-4b24-9daa-d17f7b296d8b\") " pod="openshift-dns/dns-default-kmxk8"
Oct 31 01:20:18.126943 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.126916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa56c5f6-4784-4b24-9daa-d17f7b296d8b-tmp-dir\") pod \"dns-default-kmxk8\" (UID: \"aa56c5f6-4784-4b24-9daa-d17f7b296d8b\") " pod="openshift-dns/dns-default-kmxk8"
Oct 31 01:20:18.126984 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.126946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pdph\" (UniqueName: \"kubernetes.io/projected/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-kube-api-access-7pdph\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:18.126984 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.126976    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-data-volume\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:18.127047 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.127018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60e7e4bc-764b-49e0-bb92-dea358fafcc5-cert\") pod \"ingress-canary-v9wzp\" (UID: \"60e7e4bc-764b-49e0-bb92-dea358fafcc5\") " pod="openshift-ingress-canary/ingress-canary-v9wzp"
Oct 31 01:20:18.127097 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.127048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa56c5f6-4784-4b24-9daa-d17f7b296d8b-metrics-tls\") pod \"dns-default-kmxk8\" (UID: \"aa56c5f6-4784-4b24-9daa-d17f7b296d8b\") " pod="openshift-dns/dns-default-kmxk8"
Oct 31 01:20:18.127097 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.127078    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:18.127347 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.127326    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-data-volume\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:18.127347 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.127341    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa56c5f6-4784-4b24-9daa-d17f7b296d8b-tmp-dir\") pod \"dns-default-kmxk8\" (UID: \"aa56c5f6-4784-4b24-9daa-d17f7b296d8b\") " pod="openshift-dns/dns-default-kmxk8"
Oct 31 01:20:18.127483 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.127358    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:18.127701 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.127683    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aa56c5f6-4784-4b24-9daa-d17f7b296d8b-config-volume\") pod \"dns-default-kmxk8\" (UID: \"aa56c5f6-4784-4b24-9daa-d17f7b296d8b\") " pod="openshift-dns/dns-default-kmxk8"
Oct 31 01:20:18.130485 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.130465    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa56c5f6-4784-4b24-9daa-d17f7b296d8b-metrics-tls\") pod \"dns-default-kmxk8\" (UID: \"aa56c5f6-4784-4b24-9daa-d17f7b296d8b\") " pod="openshift-dns/dns-default-kmxk8"
Oct 31 01:20:18.130578 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.130485    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:18.130752 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.130731    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60e7e4bc-764b-49e0-bb92-dea358fafcc5-cert\") pod \"ingress-canary-v9wzp\" (UID: \"60e7e4bc-764b-49e0-bb92-dea358fafcc5\") " pod="openshift-ingress-canary/ingress-canary-v9wzp"
Oct 31 01:20:18.138495 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.138471    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g8zb\" (UniqueName: \"kubernetes.io/projected/aa56c5f6-4784-4b24-9daa-d17f7b296d8b-kube-api-access-7g8zb\") pod \"dns-default-kmxk8\" (UID: \"aa56c5f6-4784-4b24-9daa-d17f7b296d8b\") " pod="openshift-dns/dns-default-kmxk8"
Oct 31 01:20:18.138585 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.138564    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c8rn\" (UniqueName: \"kubernetes.io/projected/60e7e4bc-764b-49e0-bb92-dea358fafcc5-kube-api-access-9c8rn\") pod \"ingress-canary-v9wzp\" (UID: \"60e7e4bc-764b-49e0-bb92-dea358fafcc5\") " pod="openshift-ingress-canary/ingress-canary-v9wzp"
Oct 31 01:20:18.138634 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.138605    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pdph\" (UniqueName: \"kubernetes.io/projected/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-kube-api-access-7pdph\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:18.303750 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.303686    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:18.304751 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.304730    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v9wzp"
Oct 31 01:20:18.318166 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.318131    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kmxk8"
Oct 31 01:20:18.600795 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.600724    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v9wzp"]
Oct 31 01:20:18.604089 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.603951    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kmxk8"]
Oct 31 01:20:18.604306 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.604282    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8n2r2"]
Oct 31 01:20:18.605727 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:20:18.605512    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e7e4bc_764b_49e0_bb92_dea358fafcc5.slice/crio-ce3225bd66cc30585aa3e235c6f2e0b285bbbcc7e2d207284a3ea74582082a10 WatchSource:0}: Error finding container ce3225bd66cc30585aa3e235c6f2e0b285bbbcc7e2d207284a3ea74582082a10: Status 404 returned error can't find the container with id ce3225bd66cc30585aa3e235c6f2e0b285bbbcc7e2d207284a3ea74582082a10
Oct 31 01:20:18.606620 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:20:18.606593    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa56c5f6_4784_4b24_9daa_d17f7b296d8b.slice/crio-197e8bd8725348bb6a7c30199d16b76d356a4e5051055216100b630d6bb51ae3 WatchSource:0}: Error finding container 197e8bd8725348bb6a7c30199d16b76d356a4e5051055216100b630d6bb51ae3: Status 404 returned error can't find the container with id 197e8bd8725348bb6a7c30199d16b76d356a4e5051055216100b630d6bb51ae3
Oct 31 01:20:18.607477 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:20:18.607427    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7c59f0_b861_41ab_a5ec_4f37bba8a619.slice/crio-ef2c4ca52dc80676d5f985fb9764888ef18effe4257bce45edfb81a5923c5653 WatchSource:0}: Error finding container ef2c4ca52dc80676d5f985fb9764888ef18effe4257bce45edfb81a5923c5653: Status 404 returned error can't find the container with id ef2c4ca52dc80676d5f985fb9764888ef18effe4257bce45edfb81a5923c5653
Oct 31 01:20:18.729399 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.729363    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:20:18.729399 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.729386    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:20:18.729607 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.729470    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:20:18.737271 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.737243    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 01:20:18.737410 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.737272    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 01:20:18.737522 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.737509    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-j25cw\""
Oct 31 01:20:18.737735 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.737718    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-z69qs\""
Oct 31 01:20:18.737735 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.737725    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 01:20:18.737831 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.737743    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 01:20:18.737831 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:18.737789    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lbv59\""
Oct 31 01:20:19.066059 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:19.066026    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kmxk8" event={"ID":"aa56c5f6-4784-4b24-9daa-d17f7b296d8b","Type":"ContainerStarted","Data":"197e8bd8725348bb6a7c30199d16b76d356a4e5051055216100b630d6bb51ae3"}
Oct 31 01:20:19.067394 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:19.067367    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v9wzp" event={"ID":"60e7e4bc-764b-49e0-bb92-dea358fafcc5","Type":"ContainerStarted","Data":"ce3225bd66cc30585aa3e235c6f2e0b285bbbcc7e2d207284a3ea74582082a10"}
Oct 31 01:20:19.068867 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:19.068847    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8n2r2" event={"ID":"ce7c59f0-b861-41ab-a5ec-4f37bba8a619","Type":"ContainerStarted","Data":"304ca7ceadd48e8b0406edf30c9478c86e906cf0d3aa67936b2bd5c17fc96e10"}
Oct 31 01:20:19.068948 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:19.068887    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8n2r2" event={"ID":"ce7c59f0-b861-41ab-a5ec-4f37bba8a619","Type":"ContainerStarted","Data":"ef2c4ca52dc80676d5f985fb9764888ef18effe4257bce45edfb81a5923c5653"}
Oct 31 01:20:20.073902 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:20.073869    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8n2r2" event={"ID":"ce7c59f0-b861-41ab-a5ec-4f37bba8a619","Type":"ContainerStarted","Data":"de7e4cf93d89c5ba619abc0f536b220c4371b772864d059070a81681040c961b"}
Oct 31 01:20:20.347640 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:20.347564    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-97j2h"]
Oct 31 01:20:20.349190 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:20.349174    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h"
Oct 31 01:20:20.353666 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:20.353629    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 01:20:20.353666 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:20.353656    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 31 01:20:20.353863 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:20.353748    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 01:20:20.353863 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:20.353797    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-748gw\""
Oct 31 01:20:20.353863 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:20.353802    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 01:20:20.354019 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:20.353866    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 31 01:20:20.364948 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:20.364926    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-97j2h"]
Oct 31 01:20:20.442910 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:20.442882    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-97j2h\" (UID: \"c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h"
Oct 31 01:20:20.443014 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:20.442941    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-97j2h\" (UID: \"c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h"
Oct 31 01:20:20.443014 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:20.442991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4cbn\" (UniqueName: \"kubernetes.io/projected/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243-kube-api-access-n4cbn\") pod \"prometheus-operator-9cd89b74c-97j2h\" (UID: \"c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h"
Oct 31 01:20:20.443121 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:20.443105    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-97j2h\" (UID: \"c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h"
Oct 31 01:20:20.544242 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:20.544213    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-97j2h\" (UID: \"c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h"
Oct 31 01:20:20.544402 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:20.544257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-97j2h\" (UID: \"c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h"
Oct 31 01:20:20.544402 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:20.544275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4cbn\" (UniqueName: \"kubernetes.io/projected/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243-kube-api-access-n4cbn\") pod \"prometheus-operator-9cd89b74c-97j2h\" (UID: \"c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h"
Oct 31 01:20:20.544402 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:20.544307    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-97j2h\" (UID: \"c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h"
Oct 31 01:20:20.544984 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:20.544965    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-97j2h\" (UID: \"c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h"
Oct 31 01:20:20.546725 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:20.546680    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-97j2h\" (UID: \"c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h"
Oct 31 01:20:20.546856 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:20.546833    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-97j2h\" (UID: \"c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h"
Oct 31 01:20:20.556556 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:20.556530    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4cbn\" (UniqueName: \"kubernetes.io/projected/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243-kube-api-access-n4cbn\") pod \"prometheus-operator-9cd89b74c-97j2h\" (UID: \"c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h"
Oct 31 01:20:20.662180 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:20.662105    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h"
Oct 31 01:20:20.931608 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:20.931531    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-97j2h"]
Oct 31 01:20:20.934850 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:20:20.934821    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b2a6ce_eb1d_4c15_a5cc_3a09370eb243.slice/crio-b7c02c9268fafa233a2d4682eb8d256e914786e9ccc7d60545d7e35e6ef4ff15 WatchSource:0}: Error finding container b7c02c9268fafa233a2d4682eb8d256e914786e9ccc7d60545d7e35e6ef4ff15: Status 404 returned error can't find the container with id b7c02c9268fafa233a2d4682eb8d256e914786e9ccc7d60545d7e35e6ef4ff15
Oct 31 01:20:21.077879 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:21.077841    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kmxk8" event={"ID":"aa56c5f6-4784-4b24-9daa-d17f7b296d8b","Type":"ContainerStarted","Data":"561e4119258aa0e2725ad08e34fca26fc7eff8ce16b1a9f8f5a9b5115057854a"}
Oct 31 01:20:21.078302 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:21.077896    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kmxk8" event={"ID":"aa56c5f6-4784-4b24-9daa-d17f7b296d8b","Type":"ContainerStarted","Data":"32c3f2968c0570784c0c2bc5742893164fe9d6dcc62737cf36cbdbc9625c10ba"}
Oct 31 01:20:21.078302 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:21.077959    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kmxk8"
Oct 31 01:20:21.079125 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:21.079104    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h" event={"ID":"c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243","Type":"ContainerStarted","Data":"b7c02c9268fafa233a2d4682eb8d256e914786e9ccc7d60545d7e35e6ef4ff15"}
Oct 31 01:20:21.096754 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:21.096711    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-kmxk8" podStartSLOduration=2.348376817 podStartE2EDuration="4.096698433s" totalImagesPullingTime="1.748321616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:21.096361912 +0000 UTC m=+48.953249064" watchObservedRunningTime="2026-10-31 01:20:21.096698433 +0000 UTC m=+48.953585579"
Oct 31 01:20:23.088475 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:23.088416    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8n2r2" event={"ID":"ce7c59f0-b861-41ab-a5ec-4f37bba8a619","Type":"ContainerStarted","Data":"d3ec70e4b8209b2fd7cd0fad45fb43f02318536793e38afbbad2bed26e6b2998"}
Oct 31 01:20:23.090930 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:23.090898    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h" event={"ID":"c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243","Type":"ContainerStarted","Data":"ed771431ef8ec2d641e67721813ad3ec10b4acc520a4d96ba37713d47554cff0"}
Oct 31 01:20:23.091060 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:23.090952    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h" event={"ID":"c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243","Type":"ContainerStarted","Data":"686bee7472e3ad3ad9063f093ad89bee425bc21c3168e0079359917882336a28"}
Oct 31 01:20:23.110685 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:23.110615    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8n2r2" podStartSLOduration=2.527008444 podStartE2EDuration="6.110600531s" totalImagesPullingTime="3.583592087s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:20:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:23.110111687 +0000 UTC m=+50.966998852" watchObservedRunningTime="2026-10-31 01:20:23.110600531 +0000 UTC m=+50.967487683"
Oct 31 01:20:23.130949 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:23.130890    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h" podStartSLOduration=1.378199898 podStartE2EDuration="3.13087117s" totalImagesPullingTime="1.752671272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:23.130296085 +0000 UTC m=+50.987183237" watchObservedRunningTime="2026-10-31 01:20:23.13087117 +0000 UTC m=+50.987758323"
Oct 31 01:20:24.704815 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.704781    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cdkr4"]
Oct 31 01:20:24.708362 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.708333    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.733795 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.733766    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 01:20:24.734162 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.734124    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 01:20:24.734307 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.734159    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-44nrr\""
Oct 31 01:20:24.778170 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.778138    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de2a3af0-1220-4a82-a77a-772f99694285-root\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.778363 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.778200    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de2a3af0-1220-4a82-a77a-772f99694285-metrics-client-ca\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.778363 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.778231    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.778363 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.778259    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-accelerators-collector-config\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.778363 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.778286    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzjfj\" (UniqueName: \"kubernetes.io/projected/de2a3af0-1220-4a82-a77a-772f99694285-kube-api-access-qzjfj\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.778363 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.778343    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de2a3af0-1220-4a82-a77a-772f99694285-sys\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.778642 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.778366    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-textfile\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.778642 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.778401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-tls\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.778642 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.778424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-wtmp\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.786870 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.786842    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 01:20:24.879339 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.879305    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de2a3af0-1220-4a82-a77a-772f99694285-metrics-client-ca\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.879339 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.879342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.879582 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.879364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-accelerators-collector-config\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.879582 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.879497    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzjfj\" (UniqueName: \"kubernetes.io/projected/de2a3af0-1220-4a82-a77a-772f99694285-kube-api-access-qzjfj\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.879682 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.879597    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de2a3af0-1220-4a82-a77a-772f99694285-sys\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.879682 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.879629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-textfile\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.879682 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.879670    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-tls\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.879824 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.879694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-wtmp\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.879824 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.879722    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de2a3af0-1220-4a82-a77a-772f99694285-sys\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.879824 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.879747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de2a3af0-1220-4a82-a77a-772f99694285-root\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.879950 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.879823    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de2a3af0-1220-4a82-a77a-772f99694285-root\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.880005 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.879947    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-wtmp\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.880005 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.879960    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-accelerators-collector-config\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.880084 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.880017    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de2a3af0-1220-4a82-a77a-772f99694285-metrics-client-ca\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.880084 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.880061    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-textfile\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.881664 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.881637    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.881795 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.881777    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-tls\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:24.890732 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:24.890703    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzjfj\" (UniqueName: \"kubernetes.io/projected/de2a3af0-1220-4a82-a77a-772f99694285-kube-api-access-qzjfj\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:25.025436 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.025396    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:25.038429 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:20:25.038392    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde2a3af0_1220_4a82_a77a_772f99694285.slice/crio-2f2a0313973840437025dbb2eb884792953444175adb06ecbcb4f23a435c91a6 WatchSource:0}: Error finding container 2f2a0313973840437025dbb2eb884792953444175adb06ecbcb4f23a435c91a6: Status 404 returned error can't find the container with id 2f2a0313973840437025dbb2eb884792953444175adb06ecbcb4f23a435c91a6
Oct 31 01:20:25.098813 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.098774    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v9wzp" event={"ID":"60e7e4bc-764b-49e0-bb92-dea358fafcc5","Type":"ContainerStarted","Data":"2ebf60ded26c59ccf28b513bdc88dfa62230499b88f45ecab67fe36cc3bf536e"}
Oct 31 01:20:25.100081 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.100055    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cdkr4" event={"ID":"de2a3af0-1220-4a82-a77a-772f99694285","Type":"ContainerStarted","Data":"2f2a0313973840437025dbb2eb884792953444175adb06ecbcb4f23a435c91a6"}
Oct 31 01:20:25.116042 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.115986    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-v9wzp" podStartSLOduration=1.808882751 podStartE2EDuration="8.115967715s" totalImagesPullingTime="6.307084964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:25.115180502 +0000 UTC m=+52.972067654" watchObservedRunningTime="2026-10-31 01:20:25.115967715 +0000 UTC m=+52.972854868"
Oct 31 01:20:25.881574 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.881535    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 01:20:25.885107 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.885083    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:25.889576 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.889549    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 01:20:25.889576 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.889567    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 01:20:25.889747 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.889609    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 01:20:25.889747 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.889638    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 01:20:25.889747 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.889727    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 01:20:25.889747 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.889735    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 01:20:25.889896 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.889792    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 01:20:25.889896 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.889848    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 01:20:25.889987 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.889917    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-j7vz8\""
Oct 31 01:20:25.890074 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.890059    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 01:20:25.903355 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.903266    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 01:20:25.990079 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.990037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:25.990265 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.990092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:25.990265 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.990189    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:25.990265 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.990249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:25.990414 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.990302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:25.990494 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.990438    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:25.990544 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.990506    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-config-volume\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:25.990544 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.990539    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:25.990634 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.990559    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt84f\" (UniqueName: \"kubernetes.io/projected/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-kube-api-access-qt84f\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:25.990634 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.990594    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:25.990728 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.990693    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:25.990776 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.990746    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-config-out\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:25.990822 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:25.990775    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-web-config\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.091716 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.091676    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.091969 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.091723    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.091969 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.091753    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.091969 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.091774    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.091969 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.091807    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.091969 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.091850    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.091969 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.091883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-config-volume\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.091969 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.091922    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.091969 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.091942    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt84f\" (UniqueName: \"kubernetes.io/projected/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-kube-api-access-qt84f\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.092395 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.091997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.092395 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.092034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.092395 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.092062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-config-out\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.092395 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.092089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-web-config\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.092395 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.092178    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.092658 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.092510    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.093863 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.093764    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.095790 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.095765    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.096057 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.096030    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.096356 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.096330    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-config-out\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.096514 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.096491    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.096939 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.096912    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-web-config\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.097265 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.097244    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.097782 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.097765    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.097955 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.097938    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.098918 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.098897    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-config-volume\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.104930 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.104899    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt84f\" (UniqueName: \"kubernetes.io/projected/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-kube-api-access-qt84f\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.215916 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.215882    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:26.502999 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.502967    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 01:20:26.510560 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:20:26.510531    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5d0ff7_7a5e_4a8e_916d_26e9c9572d80.slice/crio-a0b77182a4baaaa91fe5f386a9cf4c77cb619179e196a9f7fdd07b82287aae76 WatchSource:0}: Error finding container a0b77182a4baaaa91fe5f386a9cf4c77cb619179e196a9f7fdd07b82287aae76: Status 404 returned error can't find the container with id a0b77182a4baaaa91fe5f386a9cf4c77cb619179e196a9f7fdd07b82287aae76
Oct 31 01:20:26.709786 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.709751    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-86d478dd89-l66f7"]
Oct 31 01:20:26.713025 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.713007    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.720311 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.720284    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 31 01:20:26.720311 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.720297    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 31 01:20:26.720541 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.720320    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 31 01:20:26.720541 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.720336    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 31 01:20:26.720541 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.720360    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 31 01:20:26.720541 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.720342    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-83h262jp6g4hf\""
Oct 31 01:20:26.720541 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.720412    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-cg6pk\""
Oct 31 01:20:26.727320 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.727290    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-86d478dd89-l66f7"]
Oct 31 01:20:26.797757 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.797659    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76b7d8bd-5205-4243-a45f-c208a65bbc30-tmp\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.797757 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.797712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.797991 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.797841    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-tls\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.797991 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.797885    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76b7d8bd-5205-4243-a45f-c208a65bbc30-metrics-client-ca\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.797991 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.797940    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.797991 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.797970    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.798113 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.798022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdzbp\" (UniqueName: \"kubernetes.io/projected/76b7d8bd-5205-4243-a45f-c208a65bbc30-kube-api-access-fdzbp\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.798142 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.798119    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-grpc-tls\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.798172 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.798156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.898523 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.898474    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.898983 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.898545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-tls\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.898983 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.898565    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76b7d8bd-5205-4243-a45f-c208a65bbc30-metrics-client-ca\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.898983 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.898592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.898983 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.898609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.898983 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.898627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdzbp\" (UniqueName: \"kubernetes.io/projected/76b7d8bd-5205-4243-a45f-c208a65bbc30-kube-api-access-fdzbp\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.898983 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.898655    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-grpc-tls\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.898983 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.898787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.898983 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.898842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76b7d8bd-5205-4243-a45f-c208a65bbc30-tmp\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.899374 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.899256    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76b7d8bd-5205-4243-a45f-c208a65bbc30-tmp\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.899461 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.899422    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76b7d8bd-5205-4243-a45f-c208a65bbc30-metrics-client-ca\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.901575 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.901543    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.901692 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.901578    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.901692 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.901617    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.901812 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.901793    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-tls\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.901893 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.901876    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.901943 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.901900    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-grpc-tls\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:26.914666 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:26.914641    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdzbp\" (UniqueName: \"kubernetes.io/projected/76b7d8bd-5205-4243-a45f-c208a65bbc30-kube-api-access-fdzbp\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:27.036193 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:27.036143    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:27.108274 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:27.108239    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80","Type":"ContainerStarted","Data":"a0b77182a4baaaa91fe5f386a9cf4c77cb619179e196a9f7fdd07b82287aae76"}
Oct 31 01:20:27.109834 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:27.109801    2578 generic.go:356] "Generic (PLEG): container finished" podID="de2a3af0-1220-4a82-a77a-772f99694285" containerID="d8ef985f4efcdbcab7da1dd49209ef13764d3dfc281b3b45dfd65838f98926b9" exitCode=0
Oct 31 01:20:27.109966 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:27.109858    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cdkr4" event={"ID":"de2a3af0-1220-4a82-a77a-772f99694285","Type":"ContainerDied","Data":"d8ef985f4efcdbcab7da1dd49209ef13764d3dfc281b3b45dfd65838f98926b9"}
Oct 31 01:20:27.319354 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:27.319308    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-86d478dd89-l66f7"]
Oct 31 01:20:27.321695 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:20:27.321664    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b7d8bd_5205_4243_a45f_c208a65bbc30.slice/crio-eeb3287803cb703ffb46992869dd163fed0f1073887d3f04f9c2ae218331521e WatchSource:0}: Error finding container eeb3287803cb703ffb46992869dd163fed0f1073887d3f04f9c2ae218331521e: Status 404 returned error can't find the container with id eeb3287803cb703ffb46992869dd163fed0f1073887d3f04f9c2ae218331521e
Oct 31 01:20:28.114986 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:28.114899    2578 generic.go:356] "Generic (PLEG): container finished" podID="0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80" containerID="a7a3fd5806b8c7341e03e5d8e00b14a4c9a08c6f585024689294b8f1c8d663e2" exitCode=0
Oct 31 01:20:28.115426 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:28.114997    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80","Type":"ContainerDied","Data":"a7a3fd5806b8c7341e03e5d8e00b14a4c9a08c6f585024689294b8f1c8d663e2"}
Oct 31 01:20:28.117049 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:28.117025    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7" event={"ID":"76b7d8bd-5205-4243-a45f-c208a65bbc30","Type":"ContainerStarted","Data":"eeb3287803cb703ffb46992869dd163fed0f1073887d3f04f9c2ae218331521e"}
Oct 31 01:20:28.119902 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:28.119880    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cdkr4" event={"ID":"de2a3af0-1220-4a82-a77a-772f99694285","Type":"ContainerStarted","Data":"b297e17a865ada5004c9d7f95021b68baf5c671df3a4f75c3a1623716e5e7959"}
Oct 31 01:20:28.120023 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:28.119930    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cdkr4" event={"ID":"de2a3af0-1220-4a82-a77a-772f99694285","Type":"ContainerStarted","Data":"32b9b9781507ca756280169cefb8fdb51f72407b221a70650d7a5d3af90e84dd"}
Oct 31 01:20:28.168906 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:28.168837    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cdkr4" podStartSLOduration=2.660532042 podStartE2EDuration="4.168823915s" totalImagesPullingTime="1.439995099s" totalInitContainerRuntime="68.296774ms" isStatefulPod=true podCreationTimestamp="2026-10-31 01:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:28.168536116 +0000 UTC m=+56.025423271" watchObservedRunningTime="2026-10-31 01:20:28.168823915 +0000 UTC m=+56.025711067"
Oct 31 01:20:29.034836 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:29.034802    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:30.130203 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:30.130166    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80","Type":"ContainerStarted","Data":"ec6b5add2870ce01e29291ec17bfcedde4eddcefafd44a651d6209f9553be3a1"}
Oct 31 01:20:30.130685 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:30.130271    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80","Type":"ContainerStarted","Data":"627fd626c444b7a470e5756717da12b10b6b3083517932d06c7bb3dea572878d"}
Oct 31 01:20:30.130685 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:30.130304    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80","Type":"ContainerStarted","Data":"5fa7bda1fdbf10e7534be075e647582f54c3e5d68a549499a683768cf724570b"}
Oct 31 01:20:30.130685 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:30.130329    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80","Type":"ContainerStarted","Data":"8e2b806f41935ec08c9c7ede8c002547e2d55e8430c73317c30d8048f796db54"}
Oct 31 01:20:30.130685 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:30.130351    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80","Type":"ContainerStarted","Data":"6fd38e10f7a910ee8ae0557a7da3875b4fef22c74306a4ce7998af1eaf1e567b"}
Oct 31 01:20:30.132162 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:30.132138    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7" event={"ID":"76b7d8bd-5205-4243-a45f-c208a65bbc30","Type":"ContainerStarted","Data":"34f1be3d27c4d18e785e42839c8111265a72237ed16c2741bc1d49d46a82d465"}
Oct 31 01:20:30.132265 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:30.132196    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7" event={"ID":"76b7d8bd-5205-4243-a45f-c208a65bbc30","Type":"ContainerStarted","Data":"836aed8e603833f62beb2deeb20dcdaa913ff0df94348f6831faa9228babf612"}
Oct 31 01:20:30.132265 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:30.132223    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7" event={"ID":"76b7d8bd-5205-4243-a45f-c208a65bbc30","Type":"ContainerStarted","Data":"f9924ab2de98561ff0b3909f777dd9e2099b46c235e354d019cd293d8828fae4"}
Oct 31 01:20:31.013548 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.013509    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 01:20:31.019228 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.019194    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.024767 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.024744    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 01:20:31.024767 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.024746    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 31 01:20:31.024964 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.024809    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 31 01:20:31.025252 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.025226    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 31 01:20:31.025252 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.025241    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 31 01:20:31.027009 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.026918    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 31 01:20:31.027009 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.026928    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 31 01:20:31.027009 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.026936    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 31 01:20:31.027009 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.026970    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 31 01:20:31.027009 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.026984    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 31 01:20:31.027009 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.026986    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 31 01:20:31.027009 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.026976    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 31 01:20:31.027314 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.027067    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 31 01:20:31.027314 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.027258    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 31 01:20:31.027314 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.027294    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 31 01:20:31.027457 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.027345    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-svfbx\""
Oct 31 01:20:31.027457 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.027407    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-am3j72sv96m6q\""
Oct 31 01:20:31.038837 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.038739    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 01:20:31.085596 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.085568    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kmxk8"
Oct 31 01:20:31.136357 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.136319    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.136924 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.136899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.137085 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.137060    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.137186 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.137173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.137265 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.137254    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-web-config\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.137364 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.137352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.137477 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.137464    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-config\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.137587 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.137572    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.141082 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.137663    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.141082 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.137696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.141082 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.137750    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzk26\" (UniqueName: \"kubernetes.io/projected/8aa9f886-74c9-442f-89fc-beb5fc0bf260-kube-api-access-gzk26\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.141082 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.137812    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8aa9f886-74c9-442f-89fc-beb5fc0bf260-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.141082 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.137845    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8aa9f886-74c9-442f-89fc-beb5fc0bf260-config-out\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.141082 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.137886    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.141082 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.137918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.141082 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.137951    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.141082 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.137975    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.141082 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.138003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.141082 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.138040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.141082 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.138069    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.145766 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.145735    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7" event={"ID":"76b7d8bd-5205-4243-a45f-c208a65bbc30","Type":"ContainerStarted","Data":"a08030c9a8fc92bfd91e9f08671a7328c9ac5b3bd7bd8055b0b12d1c093fb9d3"}
Oct 31 01:20:31.146034 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.146016    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7" event={"ID":"76b7d8bd-5205-4243-a45f-c208a65bbc30","Type":"ContainerStarted","Data":"98ffdf25ba5bb2596945cd4f3088f95f37debf29fb497814c975321cfcbd8d44"}
Oct 31 01:20:31.146521 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.146484    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:31.146644 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.146555    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7" event={"ID":"76b7d8bd-5205-4243-a45f-c208a65bbc30","Type":"ContainerStarted","Data":"b37d5492de3a307cb28e31becf9302c3badebd9efc253fc080d9a24f3c444edc"}
Oct 31 01:20:31.151239 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.151217    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80","Type":"ContainerStarted","Data":"7aaf7c0cb3e7b6b66a124892fbaf5567d660d354f4994c8bf254e9209237f60f"}
Oct 31 01:20:31.172695 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.172643    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7" podStartSLOduration=2.106746435 podStartE2EDuration="5.172626017s" totalImagesPullingTime="3.065879582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:31.171397058 +0000 UTC m=+59.028284213" watchObservedRunningTime="2026-10-31 01:20:31.172626017 +0000 UTC m=+59.029513172"
Oct 31 01:20:31.204016 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.203963    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.679879551 podStartE2EDuration="6.20394717s" totalImagesPullingTime="3.511987675s" totalInitContainerRuntime="12.079944ms" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:31.203123706 +0000 UTC m=+59.060010900" watchObservedRunningTime="2026-10-31 01:20:31.20394717 +0000 UTC m=+59.060834359"
Oct 31 01:20:31.238865 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.238823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.239103 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.239083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.239227 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.239209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.239293 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.239277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.239360 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.239303    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-web-config\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.239360 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.239328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.239360 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.239354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-config\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.239526 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.239379    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.239526 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.239404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.239526 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.239494    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.239526 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.239502    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.239716 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.239555    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzk26\" (UniqueName: \"kubernetes.io/projected/8aa9f886-74c9-442f-89fc-beb5fc0bf260-kube-api-access-gzk26\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.239716 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.239619    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8aa9f886-74c9-442f-89fc-beb5fc0bf260-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.239716 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.239658    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8aa9f886-74c9-442f-89fc-beb5fc0bf260-config-out\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.239877 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.239732    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.239877 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.239759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.239877 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.239794    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.239877 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.239823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.239877 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.239847    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.240123 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.239882    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.240123 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.239911    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.240217 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.240124    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.240217 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.240196    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.244417 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.243226    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.244417 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.243231    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-config\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.244417 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.243917    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.244417 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.243977    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8aa9f886-74c9-442f-89fc-beb5fc0bf260-config-out\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.244417 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.244234    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.244417 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.244298    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-web-config\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.246261 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.245061    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.246261 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.245196    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.246261 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.245380    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8aa9f886-74c9-442f-89fc-beb5fc0bf260-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.246261 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.245786    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.246261 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.246017    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.246261 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.246226    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.247065 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.247042    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.247951 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.247915    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.248269 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.248221    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.250096 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.250069    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.252767 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.252721    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzk26\" (UniqueName: \"kubernetes.io/projected/8aa9f886-74c9-442f-89fc-beb5fc0bf260-kube-api-access-gzk26\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.348377 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.348287    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:31.641348 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:31.641171    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 01:20:31.643788 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:20:31.643759    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa9f886_74c9_442f_89fc_beb5fc0bf260.slice/crio-5756be9b2413eb4946f209442f92e79b3230e1fddd09d95ce3c77a7d5202e9d2 WatchSource:0}: Error finding container 5756be9b2413eb4946f209442f92e79b3230e1fddd09d95ce3c77a7d5202e9d2: Status 404 returned error can't find the container with id 5756be9b2413eb4946f209442f92e79b3230e1fddd09d95ce3c77a7d5202e9d2
Oct 31 01:20:32.156325 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:32.156288    2578 generic.go:356] "Generic (PLEG): container finished" podID="8aa9f886-74c9-442f-89fc-beb5fc0bf260" containerID="2eaf6d057afdd3fa6206f6a74542d19c3d2e45e4d4ecafc5ebf8e544583d3b79" exitCode=0
Oct 31 01:20:32.156806 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:32.156378    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa9f886-74c9-442f-89fc-beb5fc0bf260","Type":"ContainerDied","Data":"2eaf6d057afdd3fa6206f6a74542d19c3d2e45e4d4ecafc5ebf8e544583d3b79"}
Oct 31 01:20:32.156806 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:32.156486    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa9f886-74c9-442f-89fc-beb5fc0bf260","Type":"ContainerStarted","Data":"5756be9b2413eb4946f209442f92e79b3230e1fddd09d95ce3c77a7d5202e9d2"}
Oct 31 01:20:36.177048 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:36.177004    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa9f886-74c9-442f-89fc-beb5fc0bf260","Type":"ContainerStarted","Data":"261ca56b27c352c39c5e24f685ace63260ab69f6cb68bb7fcf706cf64f9a5ad8"}
Oct 31 01:20:36.177603 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:36.177089    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa9f886-74c9-442f-89fc-beb5fc0bf260","Type":"ContainerStarted","Data":"31906c6b2c76ef9d4582757d1a493aacb2e2d905957395fd984e45c6e57affbf"}
Oct 31 01:20:36.177603 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:36.177129    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa9f886-74c9-442f-89fc-beb5fc0bf260","Type":"ContainerStarted","Data":"883e7b8f050ac8dd7a3e5e0b9d4a862533b2793657b648afd4490d84255059f2"}
Oct 31 01:20:36.177603 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:36.177167    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa9f886-74c9-442f-89fc-beb5fc0bf260","Type":"ContainerStarted","Data":"9522a72fe88d60d5839043e1726e8dea287f826ad239cf7618b3bf0e7cc6872d"}
Oct 31 01:20:36.177603 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:36.177213    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa9f886-74c9-442f-89fc-beb5fc0bf260","Type":"ContainerStarted","Data":"eab52aaf130574b5e27eb56e866c36845aa82c183df2424b15c7f94e827488a2"}
Oct 31 01:20:36.177603 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:36.177254    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa9f886-74c9-442f-89fc-beb5fc0bf260","Type":"ContainerStarted","Data":"02388a871af0d7f193bafb817436a86095cb2bbacd9dc57e3c2c705cd2a3ec37"}
Oct 31 01:20:36.214582 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:36.214523    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.057519705 podStartE2EDuration="6.214507556s" totalImagesPullingTime="3.141706741s" totalInitContainerRuntime="15.28111ms" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:36.21358661 +0000 UTC m=+64.070473772" watchObservedRunningTime="2026-10-31 01:20:36.214507556 +0000 UTC m=+64.071394709"
Oct 31 01:20:36.348849 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:36.348807    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:37.162626 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:37.162597    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:38.411688 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:38.411645    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs\") pod \"network-metrics-daemon-vfk9h\" (UID: \"d9e6beef-017b-49a6-81a3-45b8faf145fc\") " pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:20:38.412142 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:38.411694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret\") pod \"global-pull-secret-syncer-5crfw\" (UID: \"f8449422-f48d-4df2-8548-9de1c5339fd0\") " pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:20:38.414324 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:38.414303    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 01:20:38.414428 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:38.414415    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 01:20:38.424569 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:38.424538    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret\") pod \"global-pull-secret-syncer-5crfw\" (UID: \"f8449422-f48d-4df2-8548-9de1c5339fd0\") " pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:20:38.424683 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:38.424654    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs\") pod \"network-metrics-daemon-vfk9h\" (UID: \"d9e6beef-017b-49a6-81a3-45b8faf145fc\") " pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:20:38.512302 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:38.512264    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xs45\" (UniqueName: \"kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45\") pod \"network-check-target-jn7dx\" (UID: \"2aa13b7e-bcfc-4ca6-96bd-086caf620c9c\") " pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:20:38.514839 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:38.514814    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 01:20:38.525391 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:38.525366    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 01:20:38.535897 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:38.535872    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xs45\" (UniqueName: \"kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45\") pod \"network-check-target-jn7dx\" (UID: \"2aa13b7e-bcfc-4ca6-96bd-086caf620c9c\") " pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:20:38.552238 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:38.552205    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-z69qs\""
Oct 31 01:20:38.552238 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:38.552205    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lbv59\""
Oct 31 01:20:38.556202 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:38.556184    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:20:38.556736 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:38.556722    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-j25cw\""
Oct 31 01:20:38.560059 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:38.560035    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:20:38.565013 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:38.564995    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:20:38.842097 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:38.842063    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5crfw"]
Oct 31 01:20:38.845084 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:38.845048    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vfk9h"]
Oct 31 01:20:38.846488 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:20:38.846440    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8449422_f48d_4df2_8548_9de1c5339fd0.slice/crio-5546da65d949c1745ed589223b92c19947df5bf8b7946d4d320a55cb5d8e2836 WatchSource:0}: Error finding container 5546da65d949c1745ed589223b92c19947df5bf8b7946d4d320a55cb5d8e2836: Status 404 returned error can't find the container with id 5546da65d949c1745ed589223b92c19947df5bf8b7946d4d320a55cb5d8e2836
Oct 31 01:20:38.847869 ip-10-0-133-98 kubenswrapper[2578]: W1031 01:20:38.847843    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e6beef_017b_49a6_81a3_45b8faf145fc.slice/crio-cee0eb429cb99610539937858f608b35ded58d6f2ac2b56d040f8bba7b7b9916 WatchSource:0}: Error finding container cee0eb429cb99610539937858f608b35ded58d6f2ac2b56d040f8bba7b7b9916: Status 404 returned error can't find the container with id cee0eb429cb99610539937858f608b35ded58d6f2ac2b56d040f8bba7b7b9916
Oct 31 01:20:38.847953 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:38.847927    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jn7dx"]
Oct 31 01:20:39.188466 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:39.188359    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vfk9h" event={"ID":"d9e6beef-017b-49a6-81a3-45b8faf145fc","Type":"ContainerStarted","Data":"cee0eb429cb99610539937858f608b35ded58d6f2ac2b56d040f8bba7b7b9916"}
Oct 31 01:20:39.189586 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:39.189555    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5crfw" event={"ID":"f8449422-f48d-4df2-8548-9de1c5339fd0","Type":"ContainerStarted","Data":"5546da65d949c1745ed589223b92c19947df5bf8b7946d4d320a55cb5d8e2836"}
Oct 31 01:20:39.190551 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:39.190526    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jn7dx" event={"ID":"2aa13b7e-bcfc-4ca6-96bd-086caf620c9c","Type":"ContainerStarted","Data":"16f972cb52728637abb62daf07e0ef221730ee2f44b1d9174bba586642ea3548"}
Oct 31 01:20:41.202114 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:41.202072    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vfk9h" event={"ID":"d9e6beef-017b-49a6-81a3-45b8faf145fc","Type":"ContainerStarted","Data":"47fd5f9bb00d3cd0d73ef5081c655fd46ff9a9666e5488712843db6661d3ffc8"}
Oct 31 01:20:41.202610 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:41.202144    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vfk9h" event={"ID":"d9e6beef-017b-49a6-81a3-45b8faf145fc","Type":"ContainerStarted","Data":"26d79356d0f19b15838f536d574d78e069e18436b2ab1c48a2221cd5ecdc88d2"}
Oct 31 01:20:41.222227 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:41.222171    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vfk9h" podStartSLOduration=66.785865764 podStartE2EDuration="1m8.222152567s" totalImagesPullingTime="1.436286803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:41.220997464 +0000 UTC m=+69.077884617" watchObservedRunningTime="2026-10-31 01:20:41.222152567 +0000 UTC m=+69.079039721"
Oct 31 01:20:43.213503 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:43.213438    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jn7dx" event={"ID":"2aa13b7e-bcfc-4ca6-96bd-086caf620c9c","Type":"ContainerStarted","Data":"db2481449e18507be571298c10dae7ad1cdf98f55ee10dda558c71cfee9c7ed7"}
Oct 31 01:20:43.213957 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:43.213545    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:20:43.233901 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:43.233836    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jn7dx" podStartSLOduration=66.788842848 podStartE2EDuration="1m10.233816568s" totalImagesPullingTime="3.44497372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:43.233244725 +0000 UTC m=+71.090131878" watchObservedRunningTime="2026-10-31 01:20:43.233816568 +0000 UTC m=+71.090703720"
Oct 31 01:20:43.940975 ip-10-0-133-98 kubenswrapper[2578]: I1031 01:20:43.940820    2578 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 01:20:43.940992 ip-10-0-133-98 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 01:20:43.950820 ip-10-0-133-98 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 01:20:43.951018 ip-10-0-133-98 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 01:20:43.951068 ip-10-0-133-98 systemd[1]: kubelet.service: Consumed 4.245s CPU time.
Oct 31 01:20:43.962265 ip-10-0-133-98 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 01:20:44.027200 ip-10-0-133-98 kubenswrapper[5650]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:20:44.027200 ip-10-0-133-98 kubenswrapper[5650]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 01:20:44.027200 ip-10-0-133-98 kubenswrapper[5650]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:20:44.027200 ip-10-0-133-98 kubenswrapper[5650]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:20:44.030308 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030214    5650 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:20:44.030308 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030308    5650 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:20:44.030308 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030316    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:20:44.030581 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030321    5650 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:20:44.030581 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030325    5650 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:20:44.030581 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030330    5650 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:20:44.030581 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030334    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:20:44.030581 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030338    5650 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:20:44.030581 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030342    5650 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:20:44.030581 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030345    5650 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:20:44.030581 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030349    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:20:44.030581 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030353    5650 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:20:44.030581 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030357    5650 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:20:44.030581 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030362    5650 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:20:44.030581 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030366    5650 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:20:44.030581 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030369    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:20:44.030581 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030373    5650 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:20:44.030581 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030378    5650 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:20:44.030581 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030382    5650 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:20:44.030581 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030386    5650 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:20:44.030581 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030390    5650 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:20:44.030581 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030394    5650 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:20:44.030581 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030398    5650 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:20:44.031485 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030402    5650 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:20:44.031485 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030406    5650 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:20:44.031485 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030410    5650 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:20:44.031485 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030414    5650 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:20:44.031485 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030418    5650 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:20:44.031485 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030422    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:20:44.031485 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030427    5650 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:20:44.031485 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030431    5650 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:20:44.031485 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030437    5650 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:20:44.031485 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030460    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:20:44.031485 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030466    5650 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:20:44.031485 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030471    5650 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:20:44.031485 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030476    5650 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:20:44.031485 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030480    5650 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:20:44.031485 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030485    5650 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:20:44.031485 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030489    5650 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:20:44.031485 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030494    5650 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:20:44.031485 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030498    5650 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:20:44.031485 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030503    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:20:44.032319 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030507    5650 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:20:44.032319 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030511    5650 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:20:44.032319 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030516    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:20:44.032319 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030521    5650 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:20:44.032319 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030526    5650 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:20:44.032319 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030531    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:20:44.032319 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030535    5650 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:20:44.032319 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030540    5650 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:20:44.032319 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030544    5650 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:20:44.032319 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030549    5650 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:20:44.032319 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030553    5650 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:20:44.032319 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030557    5650 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:20:44.032319 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030562    5650 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:20:44.032319 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030567    5650 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:20:44.032319 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030572    5650 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:20:44.032319 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030576    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:20:44.032319 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030580    5650 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:20:44.032319 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030584    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:20:44.032319 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030589    5650 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:20:44.032319 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030594    5650 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:20:44.033300 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030598    5650 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:20:44.033300 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030603    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:20:44.033300 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030608    5650 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:20:44.033300 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030613    5650 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:20:44.033300 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030617    5650 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:20:44.033300 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030622    5650 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:20:44.033300 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030626    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:20:44.033300 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030632    5650 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:20:44.033300 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030636    5650 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:20:44.033300 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030641    5650 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:20:44.033300 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030645    5650 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:20:44.033300 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030650    5650 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:20:44.033300 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030654    5650 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:20:44.033300 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030659    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:20:44.033300 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030665    5650 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:20:44.033300 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030670    5650 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:20:44.033300 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030674    5650 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:20:44.033300 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030679    5650 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:20:44.033300 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030683    5650 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:20:44.034167 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030688    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:20:44.034167 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030692    5650 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:20:44.034167 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030707    5650 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:20:44.034167 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030713    5650 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:20:44.034167 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030717    5650 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:20:44.034167 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030722    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:20:44.034167 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030726    5650 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:20:44.034167 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030730    5650 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:20:44.034167 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030735    5650 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:20:44.034167 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030739    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:20:44.034167 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030744    5650 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:20:44.034167 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030749    5650 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:20:44.034167 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030753    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:20:44.034167 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030759    5650 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:20:44.034167 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030763    5650 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:20:44.034167 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030767    5650 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:20:44.034167 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030772    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:20:44.034167 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030776    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:20:44.034167 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030781    5650 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:20:44.035062 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030786    5650 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:20:44.035062 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030791    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:20:44.035062 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030795    5650 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:20:44.035062 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030800    5650 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:20:44.035062 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030806    5650 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:20:44.035062 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.030811    5650 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:20:44.035062 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032180    5650 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:20:44.035062 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032194    5650 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:20:44.035062 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032203    5650 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:20:44.035062 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032208    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:20:44.035062 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032213    5650 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:20:44.035062 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032218    5650 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:20:44.035062 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032223    5650 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:20:44.035062 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032227    5650 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:20:44.035062 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032231    5650 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:20:44.035062 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032236    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:20:44.035062 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032251    5650 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:20:44.035062 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032258    5650 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:20:44.035925 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032264    5650 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:20:44.035925 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032269    5650 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:20:44.035925 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032274    5650 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:20:44.035925 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032278    5650 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:20:44.035925 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032283    5650 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:20:44.035925 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032287    5650 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:20:44.035925 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032292    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:20:44.035925 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032297    5650 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:20:44.035925 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032301    5650 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:20:44.035925 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032306    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:20:44.035925 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032310    5650 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:20:44.035925 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032315    5650 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:20:44.035925 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032319    5650 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:20:44.035925 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032323    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:20:44.035925 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032327    5650 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:20:44.035925 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032331    5650 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:20:44.035925 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032336    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:20:44.035925 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032340    5650 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:20:44.035925 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032344    5650 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:20:44.036761 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032348    5650 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:20:44.036761 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032352    5650 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:20:44.036761 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032357    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:20:44.036761 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032362    5650 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:20:44.036761 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032366    5650 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:20:44.036761 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032370    5650 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:20:44.036761 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032375    5650 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:20:44.036761 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032379    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:20:44.036761 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032383    5650 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:20:44.036761 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032387    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:20:44.036761 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032392    5650 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:20:44.036761 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032396    5650 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:20:44.036761 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032402    5650 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:20:44.036761 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032407    5650 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:20:44.036761 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032412    5650 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:20:44.036761 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032416    5650 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:20:44.036761 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032420    5650 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:20:44.036761 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032424    5650 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:20:44.036761 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032429    5650 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:20:44.036761 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032435    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:20:44.037430 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032439    5650 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:20:44.037430 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032468    5650 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:20:44.037430 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032473    5650 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:20:44.037430 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032477    5650 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:20:44.037430 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032481    5650 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:20:44.037430 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032486    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:20:44.037430 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032495    5650 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:20:44.037430 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032500    5650 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:20:44.037430 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032504    5650 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:20:44.037430 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032508    5650 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:20:44.037430 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032512    5650 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:20:44.037430 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032516    5650 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:20:44.037430 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032520    5650 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:20:44.037430 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032524    5650 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:20:44.037430 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032529    5650 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:20:44.037430 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032534    5650 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:20:44.037430 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032538    5650 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:20:44.037430 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032542    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:20:44.037430 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032546    5650 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:20:44.037430 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032551    5650 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:20:44.037983 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032556    5650 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:20:44.037983 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032562    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:20:44.037983 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032566    5650 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:20:44.037983 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032571    5650 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:20:44.037983 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032575    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:20:44.037983 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032582    5650 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:20:44.037983 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032586    5650 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:20:44.037983 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032591    5650 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:20:44.037983 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032597    5650 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:20:44.037983 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032601    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:20:44.037983 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032606    5650 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:20:44.037983 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032610    5650 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:20:44.037983 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032615    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:20:44.037983 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032620    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:20:44.037983 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032624    5650 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:20:44.037983 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032628    5650 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:20:44.037983 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032633    5650 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:20:44.037983 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032637    5650 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:20:44.037983 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032644    5650 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:20:44.037983 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032649    5650 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:20:44.038490 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032654    5650 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:20:44.038490 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032659    5650 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:20:44.038490 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032663    5650 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:20:44.038490 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032668    5650 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:20:44.038490 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032672    5650 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:20:44.038490 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032676    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:20:44.038490 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032681    5650 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:20:44.038490 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032686    5650 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:20:44.038490 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032690    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:20:44.038490 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032694    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:20:44.038490 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032699    5650 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:20:44.038490 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032703    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:20:44.038490 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032707    5650 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:20:44.038490 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032712    5650 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:20:44.038490 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.032717    5650 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:20:44.038490 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033596    5650 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:20:44.038490 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033607    5650 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:20:44.038490 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033613    5650 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:20:44.038490 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033618    5650 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:20:44.038490 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033623    5650 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:20:44.039054 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033627    5650 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:20:44.039054 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033631    5650 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:20:44.039054 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033636    5650 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:20:44.039054 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033642    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:20:44.039054 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033647    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:20:44.039054 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033651    5650 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:20:44.039054 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033656    5650 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:20:44.039054 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033660    5650 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:20:44.039054 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033665    5650 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:20:44.039054 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033669    5650 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:20:44.039054 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033675    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:20:44.039054 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033680    5650 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:20:44.039054 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033684    5650 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:20:44.039054 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033688    5650 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:20:44.039054 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033693    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:20:44.039054 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033698    5650 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:20:44.039054 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033702    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:20:44.039054 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033707    5650 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:20:44.039054 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033712    5650 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:20:44.039054 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033716    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:20:44.039567 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033720    5650 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:20:44.039567 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033725    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:20:44.039567 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033729    5650 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:20:44.039567 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033733    5650 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:20:44.039567 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033738    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:20:44.039567 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033743    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:20:44.039567 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033747    5650 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:20:44.039567 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033752    5650 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:20:44.039567 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033756    5650 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:20:44.039567 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033761    5650 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:20:44.039567 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033766    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:20:44.039567 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033771    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:20:44.039567 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033775    5650 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:20:44.039567 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033779    5650 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:20:44.039567 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033784    5650 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:20:44.039567 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033788    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:20:44.039567 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033793    5650 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:20:44.039567 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033797    5650 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:20:44.039567 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033802    5650 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:20:44.039567 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033806    5650 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:20:44.040055 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033811    5650 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:20:44.040055 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033815    5650 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:20:44.040055 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033819    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:20:44.040055 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033824    5650 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:20:44.040055 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033828    5650 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:20:44.040055 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033833    5650 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:20:44.040055 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033838    5650 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:20:44.040055 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033842    5650 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:20:44.040055 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033847    5650 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:20:44.040055 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033851    5650 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:20:44.040055 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033856    5650 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:20:44.040055 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033860    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:20:44.040055 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033865    5650 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:20:44.040055 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033869    5650 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:20:44.040055 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033873    5650 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:20:44.040055 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033878    5650 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:20:44.040055 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033893    5650 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:20:44.040055 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033901    5650 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:20:44.040055 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033907    5650 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:20:44.040562 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033913    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:20:44.040562 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033917    5650 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:20:44.040562 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033922    5650 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:20:44.040562 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033927    5650 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:20:44.040562 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033932    5650 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:20:44.040562 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033937    5650 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:20:44.040562 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033941    5650 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:20:44.040562 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033946    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:20:44.040562 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033950    5650 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:20:44.040562 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033954    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:20:44.040562 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033958    5650 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:20:44.040562 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033962    5650 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:20:44.040562 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033967    5650 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:20:44.040562 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033973    5650 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:20:44.040562 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033979    5650 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:20:44.040562 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033984    5650 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:20:44.040562 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033989    5650 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:20:44.040562 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033993    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:20:44.040562 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.033997    5650 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:20:44.041020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.034001    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:20:44.041020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.034006    5650 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:20:44.041020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.034010    5650 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:20:44.041020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.034015    5650 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:20:44.041020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.034019    5650 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:20:44.041020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.034023    5650 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:20:44.041020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.034028    5650 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:20:44.041020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.034032    5650 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:20:44.041020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.034036    5650 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:20:44.041020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.034040    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:20:44.041020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.034045    5650 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:20:44.041020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.034049    5650 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:20:44.041020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.034054    5650 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:20:44.041020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.034059    5650 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:20:44.041020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.034063    5650 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:20:44.041020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.034067    5650 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:20:44.041020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.034072    5650 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:20:44.041020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.034076    5650 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:20:44.041020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.034081    5650 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:20:44.041519 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.034085    5650 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:20:44.041519 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.034092    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:20:44.041519 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.034097    5650 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:20:44.041519 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.034101    5650 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:20:44.041519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034244    5650 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 01:20:44.041519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034257    5650 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 01:20:44.041519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034277    5650 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 01:20:44.041519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034290    5650 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 01:20:44.041519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034298    5650 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 01:20:44.041519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034303    5650 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 01:20:44.041519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034310    5650 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 01:20:44.041519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034317    5650 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 01:20:44.041519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034323    5650 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 01:20:44.041519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034328    5650 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 01:20:44.041519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034334    5650 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 01:20:44.041519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034342    5650 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 01:20:44.041519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034347    5650 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 01:20:44.041519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034354    5650 flags.go:64] FLAG: --cgroup-root=""
Oct 31 01:20:44.041519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034359    5650 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 01:20:44.041519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034364    5650 flags.go:64] FLAG: --client-ca-file=""
Oct 31 01:20:44.041519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034369    5650 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 01:20:44.041519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034374    5650 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 01:20:44.041519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034381    5650 flags.go:64] FLAG: --cluster-domain=""
Oct 31 01:20:44.042080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034385    5650 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 01:20:44.042080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034390    5650 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 01:20:44.042080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034396    5650 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 01:20:44.042080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034401    5650 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 01:20:44.042080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034414    5650 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 01:20:44.042080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034420    5650 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 01:20:44.042080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034425    5650 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 01:20:44.042080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034430    5650 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 01:20:44.042080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034435    5650 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 01:20:44.042080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034441    5650 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 01:20:44.042080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034462    5650 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 01:20:44.042080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034468    5650 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 01:20:44.042080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034473    5650 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 01:20:44.042080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034480    5650 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 01:20:44.042080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034486    5650 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 01:20:44.042080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034491    5650 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 01:20:44.042080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034496    5650 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 01:20:44.042080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034501    5650 flags.go:64] FLAG: --enable-server="true"
Oct 31 01:20:44.042080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034506    5650 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 01:20:44.042080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034514    5650 flags.go:64] FLAG: --event-burst="100"
Oct 31 01:20:44.042080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034519    5650 flags.go:64] FLAG: --event-qps="50"
Oct 31 01:20:44.042080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034528    5650 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 01:20:44.042080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034533    5650 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 01:20:44.042080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034538    5650 flags.go:64] FLAG: --eviction-hard=""
Oct 31 01:20:44.042688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034547    5650 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 01:20:44.042688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034552    5650 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 01:20:44.042688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034557    5650 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 01:20:44.042688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034563    5650 flags.go:64] FLAG: --eviction-soft=""
Oct 31 01:20:44.042688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034568    5650 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 01:20:44.042688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034573    5650 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 01:20:44.042688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034578    5650 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 01:20:44.042688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034583    5650 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 01:20:44.042688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034588    5650 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 01:20:44.042688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034593    5650 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 01:20:44.042688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034598    5650 flags.go:64] FLAG: --feature-gates=""
Oct 31 01:20:44.042688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034605    5650 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 01:20:44.042688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034610    5650 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 01:20:44.042688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034615    5650 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 01:20:44.042688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034621    5650 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 01:20:44.042688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034626    5650 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 01:20:44.042688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034631    5650 flags.go:64] FLAG: --help="false"
Oct 31 01:20:44.042688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034636    5650 flags.go:64] FLAG: --hostname-override="ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:20:44.042688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034643    5650 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 01:20:44.042688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034649    5650 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 01:20:44.042688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034654    5650 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 01:20:44.042688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034660    5650 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 01:20:44.042688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034666    5650 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 01:20:44.043293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034671    5650 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 01:20:44.043293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034676    5650 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 01:20:44.043293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034680    5650 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 01:20:44.043293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034685    5650 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 01:20:44.043293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034690    5650 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 01:20:44.043293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034695    5650 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 01:20:44.043293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034701    5650 flags.go:64] FLAG: --kube-reserved=""
Oct 31 01:20:44.043293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034706    5650 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 01:20:44.043293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034711    5650 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 01:20:44.043293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034718    5650 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 01:20:44.043293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034723    5650 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 01:20:44.043293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034728    5650 flags.go:64] FLAG: --lock-file=""
Oct 31 01:20:44.043293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034733    5650 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 01:20:44.043293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034739    5650 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 01:20:44.043293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034744    5650 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 01:20:44.043293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034753    5650 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 01:20:44.043293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034758    5650 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 01:20:44.043293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034763    5650 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 01:20:44.043293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034768    5650 flags.go:64] FLAG: --logging-format="text"
Oct 31 01:20:44.043293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034773    5650 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 01:20:44.043293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034779    5650 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 01:20:44.043293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034784    5650 flags.go:64] FLAG: --manifest-url=""
Oct 31 01:20:44.043293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034789    5650 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 01:20:44.043293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034797    5650 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 01:20:44.043293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034802    5650 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034808    5650 flags.go:64] FLAG: --max-pods="110"
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034814    5650 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034819    5650 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034825    5650 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034830    5650 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034835    5650 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034840    5650 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034845    5650 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034858    5650 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034863    5650 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034868    5650 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034873    5650 flags.go:64] FLAG: --pod-cidr=""
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034878    5650 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034885    5650 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034890    5650 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034895    5650 flags.go:64] FLAG: --port="10250"
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034902    5650 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034907    5650 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-000fec9b7531646b6"
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034912    5650 flags.go:64] FLAG: --qos-reserved=""
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034917    5650 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034922    5650 flags.go:64] FLAG: --register-node="true"
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034927    5650 flags.go:64] FLAG: --register-with-taints=""
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034934    5650 flags.go:64] FLAG: --registry-burst="10"
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034939    5650 flags.go:64] FLAG: --registry-qps="5"
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034944    5650 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 01:20:44.044129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034949    5650 flags.go:64] FLAG: --reserved-memory=""
Oct 31 01:20:44.044829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034955    5650 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 01:20:44.044829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034961    5650 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 01:20:44.044829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034965    5650 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 01:20:44.044829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034970    5650 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 01:20:44.044829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034975    5650 flags.go:64] FLAG: --runonce="false"
Oct 31 01:20:44.044829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034980    5650 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 01:20:44.044829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034986    5650 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 01:20:44.044829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034991    5650 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 01:20:44.044829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.034996    5650 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 01:20:44.044829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.035000    5650 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 01:20:44.044829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.035014    5650 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 01:20:44.044829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.035020    5650 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 01:20:44.044829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.035025    5650 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 01:20:44.044829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.035030    5650 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 01:20:44.044829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.035035    5650 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 01:20:44.044829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.035040    5650 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 01:20:44.044829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.035045    5650 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 01:20:44.044829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.035050    5650 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 01:20:44.044829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.035055    5650 flags.go:64] FLAG: --system-cgroups=""
Oct 31 01:20:44.044829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.035061    5650 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 01:20:44.044829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.035070    5650 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 01:20:44.044829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.035074    5650 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 01:20:44.044829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.035080    5650 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 01:20:44.044829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.035087    5650 flags.go:64] FLAG: --tls-min-version=""
Oct 31 01:20:44.044829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.035091    5650 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 01:20:44.045509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.035096    5650 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 01:20:44.045509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.035101    5650 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 01:20:44.045509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.035106    5650 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 01:20:44.045509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.035111    5650 flags.go:64] FLAG: --v="2"
Oct 31 01:20:44.045509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.035119    5650 flags.go:64] FLAG: --version="false"
Oct 31 01:20:44.045509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.035125    5650 flags.go:64] FLAG: --vmodule=""
Oct 31 01:20:44.045509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.035132    5650 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 01:20:44.045509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.035137    5650 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 01:20:44.045509 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035400    5650 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:20:44.045509 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035409    5650 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:20:44.045509 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035415    5650 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:20:44.045509 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035419    5650 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:20:44.045509 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035424    5650 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:20:44.045509 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035429    5650 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:20:44.045509 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035434    5650 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:20:44.045509 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035439    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:20:44.045509 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035462    5650 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:20:44.045509 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035468    5650 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:20:44.045509 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035475    5650 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:20:44.045509 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035482    5650 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:20:44.046421 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035487    5650 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:20:44.046421 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035493    5650 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:20:44.046421 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035498    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:20:44.046421 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035502    5650 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:20:44.046421 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035507    5650 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:20:44.046421 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035512    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:20:44.046421 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035519    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:20:44.046421 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035523    5650 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:20:44.046421 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035528    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:20:44.046421 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035534    5650 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:20:44.046421 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035538    5650 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:20:44.046421 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035543    5650 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:20:44.046421 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035548    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:20:44.046421 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035552    5650 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:20:44.046421 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035557    5650 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:20:44.046421 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035562    5650 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:20:44.046421 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035566    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:20:44.046421 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035571    5650 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:20:44.046421 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035575    5650 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:20:44.047140 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035580    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:20:44.047140 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035584    5650 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:20:44.047140 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035588    5650 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:20:44.047140 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035592    5650 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:20:44.047140 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035596    5650 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:20:44.047140 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035600    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:20:44.047140 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035604    5650 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:20:44.047140 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035609    5650 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:20:44.047140 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035613    5650 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:20:44.047140 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035618    5650 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:20:44.047140 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035622    5650 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:20:44.047140 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035627    5650 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:20:44.047140 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035632    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:20:44.047140 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035637    5650 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:20:44.047140 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035642    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:20:44.047140 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035646    5650 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:20:44.047140 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035651    5650 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:20:44.047140 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035655    5650 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:20:44.047140 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035659    5650 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:20:44.047851 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035666    5650 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:20:44.047851 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035670    5650 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:20:44.047851 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035675    5650 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:20:44.047851 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035681    5650 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:20:44.047851 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035686    5650 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:20:44.047851 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035691    5650 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:20:44.047851 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035695    5650 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:20:44.047851 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035699    5650 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:20:44.047851 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035704    5650 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:20:44.047851 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035708    5650 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:20:44.047851 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035713    5650 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:20:44.047851 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035717    5650 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:20:44.047851 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035722    5650 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:20:44.047851 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035727    5650 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:20:44.047851 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035731    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:20:44.047851 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035736    5650 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:20:44.047851 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035740    5650 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:20:44.047851 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035745    5650 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:20:44.047851 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035749    5650 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:20:44.047851 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035754    5650 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:20:44.048459 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035758    5650 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:20:44.048459 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035762    5650 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:20:44.048459 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035767    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:20:44.048459 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035771    5650 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:20:44.048459 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035776    5650 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:20:44.048459 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035780    5650 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:20:44.048459 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035792    5650 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:20:44.048459 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035798    5650 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:20:44.048459 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035803    5650 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:20:44.048459 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035808    5650 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:20:44.048459 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035812    5650 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:20:44.048459 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035816    5650 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:20:44.048459 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035822    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:20:44.048459 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035827    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:20:44.048459 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035832    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:20:44.048459 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035837    5650 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:20:44.048459 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035842    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:20:44.048459 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035847    5650 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:20:44.048459 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035852    5650 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:20:44.048459 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035857    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:20:44.049015 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035861    5650 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:20:44.049015 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035866    5650 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:20:44.049015 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035870    5650 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:20:44.049015 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035874    5650 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:20:44.049015 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035879    5650 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:20:44.049015 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035883    5650 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:20:44.049015 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035887    5650 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:20:44.049015 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035892    5650 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:20:44.049015 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035897    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:20:44.049015 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035901    5650 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:20:44.049015 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035906    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:20:44.049015 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035910    5650 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:20:44.049015 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035914    5650 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:20:44.049015 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035918    5650 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:20:44.049015 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035923    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:20:44.049015 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.035927    5650 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:20:44.049480 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.035976    5650 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:20:44.049480 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036037    5650 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:20:44.049480 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036045    5650 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:20:44.049480 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036055    5650 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:20:44.049480 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036061    5650 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:20:44.049480 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036066    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:20:44.049480 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036070    5650 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:20:44.049480 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036074    5650 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:20:44.049480 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036081    5650 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:20:44.049480 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036086    5650 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:20:44.049480 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036090    5650 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:20:44.049480 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036095    5650 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:20:44.049480 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036101    5650 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:20:44.049480 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036107    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:20:44.049480 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036111    5650 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:20:44.049480 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036115    5650 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:20:44.050042 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036120    5650 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:20:44.050042 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036125    5650 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:20:44.050042 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036131    5650 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:20:44.050042 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036135    5650 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:20:44.050042 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036140    5650 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:20:44.050042 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036144    5650 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:20:44.050042 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036149    5650 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:20:44.050042 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036153    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:20:44.050042 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036157    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:20:44.050042 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036162    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:20:44.050042 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036166    5650 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:20:44.050042 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036170    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:20:44.050042 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036174    5650 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:20:44.050042 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036179    5650 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:20:44.050042 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036183    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:20:44.050042 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036188    5650 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:20:44.050042 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036192    5650 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:20:44.050042 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036196    5650 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:20:44.050042 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036201    5650 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:20:44.050042 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036205    5650 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:20:44.050760 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036211    5650 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:20:44.050760 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036216    5650 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:20:44.050760 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036220    5650 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:20:44.050760 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036225    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:20:44.050760 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036231    5650 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:20:44.050760 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036235    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:20:44.050760 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036239    5650 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:20:44.050760 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036244    5650 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:20:44.050760 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036250    5650 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:20:44.050760 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036254    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:20:44.050760 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036258    5650 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:20:44.050760 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036263    5650 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:20:44.050760 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036268    5650 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:20:44.050760 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036273    5650 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:20:44.050760 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036277    5650 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:20:44.050760 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036281    5650 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:20:44.050760 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036286    5650 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:20:44.050760 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036290    5650 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:20:44.050760 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036295    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:20:44.051266 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036299    5650 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:20:44.051266 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036304    5650 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:20:44.051266 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036310    5650 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:20:44.051266 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036317    5650 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:20:44.051266 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036324    5650 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:20:44.051266 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036330    5650 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:20:44.051266 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036335    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:20:44.051266 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036340    5650 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:20:44.051266 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036345    5650 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:20:44.051266 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036349    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:20:44.051266 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036354    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:20:44.051266 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036358    5650 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:20:44.051266 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036363    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:20:44.051266 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036368    5650 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:20:44.051266 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036374    5650 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:20:44.051266 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036378    5650 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:20:44.051266 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036383    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:20:44.051266 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036389    5650 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:20:44.051266 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036394    5650 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:20:44.051813 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036399    5650 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:20:44.051813 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036404    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:20:44.051813 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036410    5650 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:20:44.051813 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036414    5650 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:20:44.051813 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036418    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:20:44.051813 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036423    5650 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:20:44.051813 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036427    5650 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:20:44.051813 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036432    5650 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:20:44.051813 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036436    5650 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:20:44.051813 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036458    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:20:44.051813 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036464    5650 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:20:44.051813 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036468    5650 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:20:44.051813 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036473    5650 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:20:44.051813 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036477    5650 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:20:44.051813 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036481    5650 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:20:44.051813 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036485    5650 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:20:44.051813 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036490    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:20:44.051813 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036494    5650 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:20:44.051813 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036499    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:20:44.052282 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036503    5650 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:20:44.052282 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036508    5650 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:20:44.052282 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036512    5650 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:20:44.052282 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036517    5650 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:20:44.052282 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036521    5650 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:20:44.052282 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036526    5650 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:20:44.052282 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036530    5650 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:20:44.052282 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036535    5650 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:20:44.052282 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036549    5650 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:20:44.052282 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036555    5650 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:20:44.052282 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036560    5650 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:20:44.052282 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036566    5650 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:20:44.052282 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036570    5650 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:20:44.052282 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.036574    5650 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:20:44.052282 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.040572    5650 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 01:20:44.052282 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.040586    5650 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 01:20:44.052282 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.040605    5650 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 01:20:44.052282 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.040610    5650 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 01:20:44.052282 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040671    5650 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:20:44.052282 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040676    5650 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:20:44.052780 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040680    5650 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:20:44.052780 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040684    5650 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:20:44.052780 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040687    5650 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:20:44.052780 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040690    5650 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:20:44.052780 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040693    5650 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:20:44.052780 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040695    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:20:44.052780 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040698    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:20:44.052780 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040702    5650 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:20:44.052780 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040704    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:20:44.052780 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040707    5650 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:20:44.052780 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040710    5650 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:20:44.052780 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040712    5650 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:20:44.052780 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040715    5650 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:20:44.052780 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040718    5650 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:20:44.052780 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040720    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:20:44.052780 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040723    5650 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:20:44.052780 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040726    5650 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:20:44.052780 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040729    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:20:44.052780 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040731    5650 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:20:44.053384 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040734    5650 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:20:44.053384 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040737    5650 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:20:44.053384 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040740    5650 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:20:44.053384 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040744    5650 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:20:44.053384 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040747    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:20:44.053384 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040750    5650 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:20:44.053384 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040753    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:20:44.053384 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040755    5650 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:20:44.053384 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040758    5650 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:20:44.053384 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040761    5650 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:20:44.053384 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040764    5650 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:20:44.053384 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040767    5650 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:20:44.053384 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040770    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:20:44.053384 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040773    5650 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:20:44.053384 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040777    5650 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:20:44.053384 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040780    5650 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:20:44.053384 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040784    5650 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:20:44.053384 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040789    5650 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:20:44.053384 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040792    5650 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:20:44.054092 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040796    5650 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:20:44.054092 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040798    5650 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:20:44.054092 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040802    5650 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:20:44.054092 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040804    5650 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:20:44.054092 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040807    5650 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:20:44.054092 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040810    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:20:44.054092 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040813    5650 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:20:44.054092 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040815    5650 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:20:44.054092 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040818    5650 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:20:44.054092 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040821    5650 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:20:44.054092 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040824    5650 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:20:44.054092 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040826    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:20:44.054092 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040838    5650 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:20:44.054092 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040841    5650 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:20:44.054092 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040844    5650 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:20:44.054092 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040847    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:20:44.054092 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040851    5650 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:20:44.054092 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040854    5650 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:20:44.054092 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040857    5650 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:20:44.054092 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040859    5650 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:20:44.054802 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040862    5650 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:20:44.054802 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040865    5650 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:20:44.054802 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040867    5650 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:20:44.054802 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040870    5650 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:20:44.054802 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040873    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:20:44.054802 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040875    5650 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:20:44.054802 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040878    5650 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:20:44.054802 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040881    5650 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:20:44.054802 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040884    5650 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:20:44.054802 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040886    5650 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:20:44.054802 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040889    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:20:44.054802 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040892    5650 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:20:44.054802 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040894    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:20:44.054802 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040897    5650 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:20:44.054802 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040900    5650 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:20:44.054802 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040903    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:20:44.054802 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040906    5650 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:20:44.054802 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040908    5650 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:20:44.054802 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040911    5650 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:20:44.055441 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040914    5650 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:20:44.055441 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040917    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:20:44.055441 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040920    5650 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:20:44.055441 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040923    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:20:44.055441 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040926    5650 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:20:44.055441 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040928    5650 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:20:44.055441 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040931    5650 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:20:44.055441 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040934    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:20:44.055441 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040937    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:20:44.055441 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040940    5650 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:20:44.055441 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040943    5650 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:20:44.055441 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040945    5650 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:20:44.055441 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040948    5650 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:20:44.055441 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040951    5650 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:20:44.055441 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040954    5650 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:20:44.055441 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040956    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:20:44.055441 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040959    5650 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:20:44.055441 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040962    5650 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:20:44.055441 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040964    5650 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:20:44.055441 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040967    5650 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:20:44.056020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040970    5650 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:20:44.056020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040972    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:20:44.056020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040975    5650 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:20:44.056020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040977    5650 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:20:44.056020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040980    5650 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:20:44.056020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040983    5650 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:20:44.056020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.040986    5650 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:20:44.056020 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.041013    5650 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:20:44.056020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041166    5650 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:20:44.056020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041172    5650 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:20:44.056020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041175    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:20:44.056020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041178    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:20:44.056020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041181    5650 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:20:44.056020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041184    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:20:44.056020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041188    5650 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:20:44.056020 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041190    5650 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:20:44.056412 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041193    5650 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:20:44.056412 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041196    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:20:44.056412 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041199    5650 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:20:44.056412 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041202    5650 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:20:44.056412 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041205    5650 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:20:44.056412 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041207    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:20:44.056412 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041210    5650 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:20:44.056412 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041214    5650 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:20:44.056412 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041216    5650 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:20:44.056412 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041219    5650 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:20:44.056412 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041222    5650 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:20:44.056412 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041225    5650 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:20:44.056412 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041227    5650 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:20:44.056412 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041230    5650 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:20:44.056412 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041233    5650 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:20:44.056412 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041235    5650 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:20:44.056412 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041238    5650 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:20:44.056412 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041241    5650 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:20:44.056412 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041243    5650 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:20:44.056412 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041246    5650 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:20:44.056916 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041249    5650 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:20:44.056916 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041251    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:20:44.056916 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041254    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:20:44.056916 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041257    5650 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:20:44.056916 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041259    5650 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:20:44.056916 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041262    5650 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:20:44.056916 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041266    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:20:44.056916 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041268    5650 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:20:44.056916 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041271    5650 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:20:44.056916 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041274    5650 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:20:44.056916 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041277    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:20:44.056916 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041280    5650 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:20:44.056916 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041282    5650 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:20:44.056916 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041285    5650 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:20:44.056916 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041289    5650 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:20:44.056916 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041292    5650 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:20:44.056916 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041296    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:20:44.056916 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041299    5650 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:20:44.056916 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041302    5650 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:20:44.057374 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041306    5650 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:20:44.057374 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041310    5650 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:20:44.057374 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041313    5650 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:20:44.057374 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041317    5650 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:20:44.057374 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041320    5650 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:20:44.057374 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041323    5650 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:20:44.057374 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041325    5650 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:20:44.057374 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041328    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:20:44.057374 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041330    5650 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:20:44.057374 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041333    5650 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:20:44.057374 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041336    5650 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:20:44.057374 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041338    5650 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:20:44.057374 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041341    5650 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:20:44.057374 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041344    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:20:44.057374 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041347    5650 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:20:44.057374 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041350    5650 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:20:44.057374 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041353    5650 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:20:44.057374 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041355    5650 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:20:44.057374 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041358    5650 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:20:44.057848 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041361    5650 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:20:44.057848 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041364    5650 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:20:44.057848 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041366    5650 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:20:44.057848 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041369    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:20:44.057848 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041372    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:20:44.057848 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041374    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:20:44.057848 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041377    5650 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:20:44.057848 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041380    5650 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:20:44.057848 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041383    5650 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:20:44.057848 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041385    5650 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:20:44.057848 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041388    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:20:44.057848 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041390    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:20:44.057848 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041393    5650 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:20:44.057848 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041396    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:20:44.057848 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041398    5650 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:20:44.057848 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041401    5650 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:20:44.057848 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041404    5650 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:20:44.057848 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041407    5650 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:20:44.057848 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041410    5650 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:20:44.058307 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041413    5650 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:20:44.058307 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041415    5650 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:20:44.058307 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041418    5650 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:20:44.058307 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041420    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:20:44.058307 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041423    5650 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:20:44.058307 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041426    5650 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:20:44.058307 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041428    5650 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:20:44.058307 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041431    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:20:44.058307 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041434    5650 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:20:44.058307 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041437    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:20:44.058307 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041440    5650 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:20:44.058307 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041455    5650 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:20:44.058307 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041458    5650 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:20:44.058307 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041461    5650 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:20:44.058307 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041464    5650 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:20:44.058307 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041467    5650 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:20:44.058307 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041470    5650 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:20:44.058307 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041472    5650 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:20:44.058307 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041475    5650 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:20:44.058307 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041478    5650 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:20:44.058796 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041480    5650 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:20:44.058796 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.041505    5650 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:20:44.058796 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041538    5650 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:20:44.058796 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041542    5650 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:20:44.058796 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041545    5650 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:20:44.058796 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041548    5650 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:20:44.058796 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041551    5650 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:20:44.058796 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041554    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:20:44.058796 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041557    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:20:44.058796 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041560    5650 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:20:44.058796 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041563    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:20:44.058796 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041566    5650 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:20:44.058796 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041569    5650 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:20:44.058796 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041572    5650 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:20:44.058796 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041575    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:20:44.058796 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041577    5650 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:20:44.059208 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041580    5650 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:20:44.059208 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041583    5650 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:20:44.059208 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041585    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:20:44.059208 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041588    5650 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:20:44.059208 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041591    5650 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:20:44.059208 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041594    5650 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:20:44.059208 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041597    5650 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:20:44.059208 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041600    5650 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:20:44.059208 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041602    5650 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:20:44.059208 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041605    5650 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:20:44.059208 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041608    5650 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:20:44.059208 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041611    5650 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:20:44.059208 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041613    5650 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:20:44.059208 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041616    5650 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:20:44.059208 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041619    5650 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:20:44.059208 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041621    5650 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:20:44.059208 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041624    5650 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:20:44.059208 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041627    5650 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:20:44.059208 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041629    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:20:44.059208 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041632    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:20:44.059707 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041635    5650 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:20:44.059707 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041638    5650 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:20:44.059707 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041641    5650 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:20:44.059707 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041644    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:20:44.059707 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041647    5650 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:20:44.059707 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041649    5650 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:20:44.059707 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041652    5650 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:20:44.059707 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041655    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:20:44.059707 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041658    5650 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:20:44.059707 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041660    5650 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:20:44.059707 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041663    5650 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:20:44.059707 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041667    5650 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:20:44.059707 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041671    5650 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:20:44.059707 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041674    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:20:44.059707 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041677    5650 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:20:44.059707 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041680    5650 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:20:44.059707 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041685    5650 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:20:44.059707 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041688    5650 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:20:44.059707 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041691    5650 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:20:44.060152 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041694    5650 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:20:44.060152 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041697    5650 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:20:44.060152 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041701    5650 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:20:44.060152 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041703    5650 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:20:44.060152 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041706    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:20:44.060152 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041709    5650 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:20:44.060152 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041711    5650 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:20:44.060152 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041714    5650 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:20:44.060152 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041716    5650 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:20:44.060152 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041719    5650 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:20:44.060152 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041722    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:20:44.060152 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041724    5650 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:20:44.060152 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041727    5650 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:20:44.060152 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041730    5650 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:20:44.060152 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041733    5650 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:20:44.060152 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041736    5650 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:20:44.060152 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041738    5650 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:20:44.060152 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041741    5650 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:20:44.060152 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041744    5650 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:20:44.060860 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041746    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:20:44.060860 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041749    5650 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:20:44.060860 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041752    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:20:44.060860 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041755    5650 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:20:44.060860 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041757    5650 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:20:44.060860 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041760    5650 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:20:44.060860 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041763    5650 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:20:44.060860 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041765    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:20:44.060860 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041768    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:20:44.060860 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041771    5650 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:20:44.060860 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041774    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:20:44.060860 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041777    5650 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:20:44.060860 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041780    5650 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:20:44.060860 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041782    5650 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:20:44.060860 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041785    5650 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:20:44.060860 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041788    5650 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:20:44.060860 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041790    5650 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:20:44.060860 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041793    5650 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:20:44.060860 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041796    5650 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:20:44.061766 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041798    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:20:44.061766 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041801    5650 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:20:44.061766 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041804    5650 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:20:44.061766 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041806    5650 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:20:44.061766 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041809    5650 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:20:44.061766 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041812    5650 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:20:44.061766 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041814    5650 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:20:44.061766 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041817    5650 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:20:44.061766 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041820    5650 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:20:44.061766 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041828    5650 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:20:44.061766 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041839    5650 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:20:44.061766 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041842    5650 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:20:44.061766 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041844    5650 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:20:44.061766 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041847    5650 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:20:44.061766 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:20:44.041850    5650 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:20:44.061766 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.041911    5650 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 01:20:44.061766 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.042924    5650 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 01:20:44.061766 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.042987    5650 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 01:20:44.061766 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.043173    5650 server.go:1014] "Starting client certificate rotation"
Oct 31 01:20:44.062550 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.043302    5650 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 01:20:44.062550 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.043357    5650 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 01:14:33 +0000 UTC" deadline="2028-07-09 18:11:27.712573134 +0000 UTC"
Oct 31 01:20:44.062550 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.043386    5650 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14824h50m43.669188868s"
Oct 31 01:20:44.062550 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.043728    5650 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 01:20:44.062550 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.044836    5650 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 01:20:44.062550 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.047230    5650 log.go:25] "Validated CRI v1 runtime API"
Oct 31 01:20:44.062550 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.049325    5650 log.go:25] "Validated CRI v1 image API"
Oct 31 01:20:44.062550 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.050136    5650 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 01:20:44.062550 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.052384    5650 fs.go:135] Filesystem UUIDs: map[3b3b78fb-536b-40e5-8c9c-848963575470:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9cea673a-776a-431e-ad60-8fa157122841:/dev/nvme0n1p4]
Oct 31 01:20:44.062988 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.052406    5650 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0fd8f4b05e04837e0e0b26e5608e81495837e6095d72ec6fad2bef34dcaf7974/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0fd8f4b05e04837e0e0b26e5608e81495837e6095d72ec6fad2bef34dcaf7974/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/167a4eb856c92f9ac3fc51b87c43d6bca9a72a347aae3bf431c7b8203a9de001/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/167a4eb856c92f9ac3fc51b87c43d6bca9a72a347aae3bf431c7b8203a9de001/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16f972cb52728637abb62daf07e0ef221730ee2f44b1d9174bba586642ea3548/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16f972cb52728637abb62daf07e0ef221730ee2f44b1d9174bba586642ea3548/userdata/shm major:0 minor:426 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/197e8bd8725348bb6a7c30199d16b76d356a4e5051055216100b630d6bb51ae3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/197e8bd8725348bb6a7c30199d16b76d356a4e5051055216100b630d6bb51ae3/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1bc364fd433ed3d2f63c9112e2d37ed896872d6d972a9b39d31c407825d5bd70/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1bc364fd433ed3d2f63c9112e2d37ed896872d6d972a9b39d31c407825d5bd70/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2a7902ff147c883af26f277241b0fa036f5e800913f8d7aeb1709878f1d15002/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2a7902ff147c883af26f277241b0fa036f5e800913f8d7aeb1709878f1d15002/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f2a0313973840437025dbb2eb884792953444175adb06ecbcb4f23a435c91a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f2a0313973840437025dbb2eb884792953444175adb06ecbcb4f23a435c91a6/userdata/shm major:0 minor:267 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f82d934690d4d27dd037f936a6e4c17907e23379df72d3eb8ba44e6d2c6d442/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f82d934690d4d27dd037f936a6e4c17907e23379df72d3eb8ba44e6d2c6d442/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5546da65d949c1745ed589223b92c19947df5bf8b7946d4d320a55cb5d8e2836/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5546da65d949c1745ed589223b92c19947df5bf8b7946d4d320a55cb5d8e2836/userdata/shm major:0 minor:423 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5756be9b2413eb4946f209442f92e79b3230e1fddd09d95ce3c77a7d5202e9d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5756be9b2413eb4946f209442f92e79b3230e1fddd09d95ce3c77a7d5202e9d2/userdata/shm major:0 minor:384 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5da57e3fdb12e9c5a9f6d66de998f92b009e424cd0f4f55a19e8644e439d3c93/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5da57e3fdb12e9c5a9f6d66de998f92b009e424cd0f4f55a19e8644e439d3c93/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/801d38c689ae4e6ea6e4a69f864ddc10453c1012b70f6632a1dfe0a7496e7659/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/801d38c689ae4e6ea6e4a69f864ddc10453c1012b70f6632a1dfe0a7496e7659/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9849baee4ffe03763bf3fa9a1bc91f2b9d3a2c828d6cb0d18680c50d8505efa7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9849baee4ffe03763bf3fa9a1bc91f2b9d3a2c828d6cb0d18680c50d8505efa7/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a0b77182a4baaaa91fe5f386a9cf4c77cb619179e196a9f7fdd07b82287aae76/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a0b77182a4baaaa91fe5f386a9cf4c77cb619179e196a9f7fdd07b82287aae76/userdata/shm major:0 minor:285 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5d3ecbcd60a975ff2c124eb66e9c9f22998bce80d5d69ad3359575bf6360e19/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5d3ecbcd60a975ff2c124eb66e9c9f22998bce80d5d69ad3359575bf6360e19/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7c02c9268fafa233a2d4682eb8d256e914786e9ccc7d60545d7e35e6ef4ff15/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7c02c9268fafa233a2d4682eb8d256e914786e9ccc7d60545d7e35e6ef4ff15/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce3225bd66cc30585aa3e235c6f2e0b285bbbcc7e2d207284a3ea74582082a10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce3225bd66cc30585aa3e235c6f2e0b285bbbcc7e2d207284a3ea74582082a10/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cee0eb429cb99610539937858f608b35ded58d6f2ac2b56d040f8bba7b7b9916/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cee0eb429cb99610539937858f608b35ded58d6f2ac2b56d040f8bba7b7b9916/userdata/shm major:0 minor:422 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eeb3287803cb703ffb46992869dd163fed0f1073887d3f04f9c2ae218331521e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eeb3287803cb703ffb46992869dd163fed0f1073887d3f04f9c2ae218331521e/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef2c4ca52dc80676d5f985fb9764888ef18effe4257bce45edfb81a5923c5653/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef2c4ca52dc80676d5f985fb9764888ef18effe4257bce45edfb81a5923c5653/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f62e7b940902c55bf18ce7cbe90f35be40ea9216ce7ade5ee556fcac904d1e55/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f62e7b940902c55bf18ce7cbe90f35be40ea9216ce7ade5ee556fcac904d1e55/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02c05e0c-b77a-45e4-b49b-1bed7e7588d0/volumes/kubernetes.io~projected/kube-api-access-v74ww:{mountpoint:/var/lib/kubelet/pods/02c05e0c-b77a-45e4-b49b-1bed7e7588d0/volumes/kubernetes.io~projected/kube-api-access-v74ww major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volume-subpaths/web-config/alertmanager/9 major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volume-subpaths/web-config/config-reloader/6 major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~empty-dir/config-out major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~projected/kube-api-access-qt84f:{mountpoint:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~projected/kube-api-access-qt84f major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~projected/tls-assets major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~secret/config-volume major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~secret/web-config major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c/volumes/kubernetes.io~projected/kube-api-access-9xs45:{mountpoint:/var/lib/kubelet/pods/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c/volumes/kubernetes.io~projected/kube-api-access-9xs45 major:0 minor:421 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52e25fa9-6c04-453a-a0b2-cca32a428a5a/volumes/kubernetes.io~projected/kube-api-access-nw5bx:{mountpoint:/var/lib/kubelet/pods/52e25fa9-6c04-453a-a0b2-cca32a428a5a/volumes/kubernetes.io~projected/kube-api-access-nw5bx major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60e7e4bc-764b-49e0-bb92-dea358fafcc5/volumes/kubernetes.io~projected/kube-api-access-9c8rn:{mountpoint:/var/lib/kubelet/pods/60e7e4bc-764b-49e0-bb92-dea358fafcc5/volumes/kubernetes.io~projected/kube-api-access-9c8rn major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60e7e4bc-764b-49e0-bb92-dea358fafcc5/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/60e7e4bc-764b-49e0-bb92-dea358fafcc5/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b7841e1-63bb-4e71-972e-1020c47d2ccb/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/6b7841e1-63bb-4e71-972e-1020c47d2ccb/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b7841e1-63bb-4e71-972e-1020c47d2ccb/volumes/kubernetes.io~projected/kube-api-access-kh6l2:{mountpoint:/var/lib/kubelet/pods/6b7841e1-63bb-4e71-972e-1020c47d2ccb/volumes/kubernetes.io~projected/kube-api-access-kh6l2 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b7841e1-63bb-4e71-972e-1020c47d2ccb/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/6b7841e1-63bb-4e71-972e-1020c47d2ccb/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76b7d8bd-5205-4243-a45f-c208a65bbc30/volumes/kubernetes.io~projected/kube-api-access-fdzbp:{mountpoint:/var/lib/kubelet/pods/76b7d8bd-5205-4243-a45f-c208a65bbc30/volumes/kubernetes.io~projected/kube-api-access-fdzbp major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76b7d8bd-5205-4243-a45f-c208a65bbc30/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/76b7d8bd-5205-4243-a45f-c208a65bbc30/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76b7d8bd-5205-4243-a45f-c208a65bbc30/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/76b7d8bd-5205-4243-a45f-c208a65bbc30/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76b7d8bd-5205-4243-a45f-c208a65bbc30/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/76b7d8bd-5205-4243-a45f-c208a65bbc30/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76b7d8bd-5205-4243-a45f-c208a65bbc30/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/76b7d8bd-5205-4243-a45f-c208a65bbc30/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76b7d8bd-5205-4243-a45f-c208a65bbc30/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/76b7d8bd-5205-4243-a45f-c208a65bbc30/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76b7d8bd-5205-4243-a45f-c208a65bbc30/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/76b7d8bd-5205-4243-a45f-c208a65bbc30/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81c10625-0368-49c7-89cf-eb51f06005d0/volumes/kubernetes.io~projected/kube-api-access-v8tn8:{mountpoint:/var/lib/kubelet/pods/81c10625-0368-49c7-89cf-eb51f06005d0/volumes/kubernetes.io~projected/kube-api-access-v8tn8 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86947040-78ac-43d5-b932-9acd63cf43e7/volumes/kubernetes.io~projected/kube-api-access-v9jkq:{mountpoint:/var/lib/kubelet/pods/86947040-78ac-43d5-b932-9acd63cf43e7/volumes/kubernetes.io~projected/kube-api-access-v9jkq major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volume-subpaths/web-config/config-reloader/2 major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volume-subpaths/web-config/prometheus/15 major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~empty-dir/config-out major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~projected/kube-api-access-gzk26:{mountpoint:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~projected/kube-api-access-gzk26 major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~projected/tls-assets major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/config major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:366 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:379 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:380 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/web-config major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa56c5f6-4784-4b24-9daa-d17f7b296d8b/volumes/kubernetes.io~projected/kube-api-access-7g8zb:{mountpoint:/var/lib/kubelet/pods/aa56c5f6-4784-4b24-9daa-d17f7b296d8b/volumes/kubernetes.io~projected/kube-api-access-7g8zb major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa56c5f6-4784-4b24-9daa-d17f7b296d8b/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/aa56c5f6-4784-4b24-9daa-d17f7b296d8b/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0cbe408-97f0-48da-ab7d-ab4d144f0796/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b0cbe408-97f0-48da-ab7d-ab4d144f0796/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0cbe408-97f0-48da-ab7d-ab4d144f0796/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b0cbe408-97f0-48da-ab7d-ab4d144f0796/volumes/kubernetes.io~empty-dir/tmp major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0cbe408-97f0-48da-ab7d-ab4d144f0796/volumes/kubernetes.io~projected/kube-api-access-xmwzw:{mountpoint:/var/lib/kubelet/pods/b0cbe408-97f0-48da-ab7d-ab4d144f0796/volumes/kubernetes.io~projected/kube-api-access-xmwzw major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243/volumes/kubernetes.io~projected/kube-api-access-n4cbn:{mountpoint:/var/lib/kubelet/pods/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243/volumes/kubernetes.io~projected/kube-api-access-n4cbn major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce7c59f0-b861-41ab-a5ec-4f37bba8a619/volumes/kubernetes.io~projected/kube-api-access-7pdph:{mountpoint:/var/lib/kubelet/pods/ce7c59f0-b861-41ab-a5ec-4f37bba8a619/volumes/kubernetes.io~projected/kube-api-access-7pdph major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce7c59f0-b861-41ab-a5ec-4f37bba8a619/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/ce7c59f0-b861-41ab-a5ec-4f37bba8a619/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9e6beef-017b-49a6-81a3-45b8faf145fc/volumes/kubernetes.io~projected/kube-api-access-5md27:{mountpoint:/var/lib/kubelet/pods/d9e6beef-017b-49a6-81a3-45b8faf145fc/volumes/kubernetes.io~projected/kube-api-access-5md27 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9e6beef-017b-49a6-81a3-45b8faf145fc/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/d9e6beef-017b-49a6-81a3-45b8faf145fc/volumes/kubernetes.io~secret/metrics-certs major:0 minor:416 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db94c614-bfe5-4328-b1aa-770694e4fca1/volumes/kubernetes.io~projected/kube-api-access-xwd6x:{mountpoint:/var/lib/kubelet/pods/db94c614-bfe5-4328-b1aa-770694e4fca1/volumes/kubernetes.io~projected/kube-api-access-xwd6x major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de2a3af0-1220-4a82-a77a-772f99694285/volumes/kubernetes.io~projected/kube-api-access-qzjfj:{mountpoint:/var/lib/kubelet/pods/de2a3af0-1220-4a82-a77a-772f99694285/volumes/kubernetes.io~projected/kube-api-access-qzjfj major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de2a3af0-1220-4a82-a77a-772f99694285/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/de2a3af0-1220-4a82-a77a-772f99694285/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de2a3af0-1220-4a82-a77a-772f99694285/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/de2a3af0-1220-4a82-a77a-772f99694285/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec3e19fc-4890-424b-86ef-f5ad1e926158/volumes/kubernetes.io~projected/kube-api-access-qr95h:{mountpoint:/var/lib/kubelet/pods/ec3e19fc-4890-424b-86ef-f5ad1e926158/volumes/kubernetes.io~projected/kube-api-access-qr95h major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8449422-f48d-4df2-8548-9de1c5339fd0/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/f8449422-f48d-4df2-8548-9de1c5339fd0/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:420 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/d5e7a698d3b1f8f8b338474c62d30b2c3962f84160d7bc5118355c1c9252edf1/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/9f41480b28baeafb042f806f916b5133b1e448f984468974644aa32adf49e632/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/c5e49f9131ce66cfd028f227d85f28b6d81dc577ce3ab4e68048531270746d97/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/40dffe15b998041c2867ddecdcd9df2f649471b1bd2d0ef83430cb85f1f6c1e0/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/14f83e6108abca6c0d60a0452882bed18e2646b14e0ad6dd36db44523fdb1af1/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/e4899d2bda1414b481cf4600f4eaaa3c831fb354d2a9f61510918b0dae2cc75a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/71293d296ea3f617bab4502aee616fcd81753a8fd114b206af4656b97fd9ba66/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b61f85ee08ca6cfc8693112bf06e453dfe8a95b8e904212225f252baa4bd0da4/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/ae95b6379b9fd38b7315f8b23044ed498de6a22eb3a08275471211cf1e730d79/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/82dd3bd3d700a75e159002c673a6d493ce859b75301179a1dd42f3825b1bd652/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/455c85da25352b539cc29a235bea5f986a25735647553e4b83f7babe058e1d35/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/928e76e15e8b2ffc781f278da3d8f05a98dc9e5d9477407b5d9c6bcbab03a948/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/7a64b14238cc8e15c24a2910e1ca8c407fbd1d600f3d2ff42c78019c6b421753/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/8a4b6d446efdbc63dcefdc9295f3c8a7cdf200491c091e354c182b1ab3dec214/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-158:{mountpoint:/var/lib/containers/storage/overlay/b2a00a56b23be4170baf96c31c7d6721fd9d55767e63f9494a4f8738b597fa9a/merged major:0 minor:158 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/68e0951e3c8b24f8cf672f0f3bce6e2d92e5a023cbde5729f5db283b1d8027ae/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/bd94e1cec5b8b1f36ad1f759ce7ca7f5bf35962b07f6381bb78bc16a6495603e/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d74cfa435d33ee36f506001a635c429a9fb48906b42db93682f04f9068ccecf1/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/3c081be6caa8606f5eec3c0025feafd95f17cf69fd50c27abfa405403bc4bd2a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/9472df2c09633a763d6198e8d578e226a932ec555f1c72e1899014c3ebca763f/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/c564385df005fd0c71556286ba9c73ee968169c9fcfaad9644ba4808d11fd457/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/964d95f6ca8493760c44474c87450f26b4958915d72d5e06384202f0e911733c/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/dfa12f7c4f3ec921aeaf463db49a45e750ce4e5b3f05ce15cc7ef0a933f46fd5/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/26b6bf125193d5655b10d0fca1324d65df00e935506637cc0062513e0a849a2c/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/8193c6a4a953d43559ed4445c8674936feef2d563a79665227fceed14d7903da/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/3b20a6ceaa29aadfced964c810ac4c74901a456764d73b8ca9f3ab938adfc503/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/10c0a196571f7e8a9af69011e67db33283f71e15131e2367b619f13c96196761/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/d36a0a90c79a2b08e2c30ad4dd33de99fa03837333eac3119e10d7fa50622377/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/92e845392976c819f95c8c3b473c964c77414702b6b696f61caa734c3bb3c201/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/bab3c01ef2ab07d33d6447b6ac1a7784b83dd9d62184f4e1076fa9a4154d5482/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/40627ef8b1b3073856ce72ead62ff7687804229b489a9e7a81aac561fa774c60/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/923fb172e7ad3a40451042267542998bc80b73160235581b0f1e9048200e9ca2/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/5db81dd3cd196231aeb13739fc8f99de0e8af0e0ecdbbd72af1ed8b986b7e18c/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/85165cf2bfba20b5e007d14a2a968bccce9beed13f0c407def16335ed7264aeb/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/e67d6fa775f9f68d7fdd19e5944b4a311da2551e19ee6cbf5948d41b16fa7c81/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/a825a482d45b507e4fb0df99e276fd06ef11dded2d8ab23e2a8b5fb4dc0974b8/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/aa1dc86e2ec7099d226e3f281844c0c27f1ce4397ecb60457936d7b0b28cb863/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/89749f43fad351a04d1b71b98ac2ed1c7df5119dabaa205e4d92538b0eaaf620/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/9b376cb1709683ff8dd230eb21e3176c4c31d3051eca15821b71ce291056b247/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/d05a4927ef01aef58efb9917aeac39b3b960c00629016d60ea587f1d7db8b5ee/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/9dc1a867c472515fb6823adff99c6911fcd19dff342541b9d76f769fc9baa776/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/f3c0fc8d041a11175ca3f1949064cd9c27ddacc7612d3d920fed8d54c98382c4/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/fbbf3afddb3f0971eaddcfea387a4e8e5792f8160b537889844748ece4d82e25/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/baaa11e0e1082f858b2ded4ba04290d6e243db7aa51ffd94944c262162eacc22/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/dd8cdf0da2c8b81ca197dec3d370938268b9fd4034c22b6e7f126dade56b7bd1/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/6983fc1bf1c1728403df44d94378433673cd5127cd84755b47270db90d4d235a/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/4f97f061dec3bff7d8cde04ce22c6d84925de5957dab080e3688e613a9125567/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/65af7066178d82950be5fd242b4cd35ab0a3d1a7e8e445b79f8d3fb07ce34a1b/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/f085da05dd2606b6025bc498161140472562a25c618fa6b09caa13738a06b681/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/e6a28465fdd88d709e7ceb19f6f35e7f159474cc70b505feb1caf737b7347803/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/b3d67a08a0c48f74125ffa652960a0afb93b1ce84409663ed5ebbf1a3c320a7f/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/6b7fe749edf5fbf80c6e99fba4fb14754b43c276d07d9bd666c76f3ca3439cec/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/6dc7d4d83fcedc78f828a924947020e6ead2bfc132a2c40eba9fb2f532e9918a/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/56e65d2b31f3f420929c8af2080cc765f755dc1e03ce5c168bef4af9733549ab/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/944cb1e0b777aaa690f988a9130b74069fa8f4ce2aa607e3d88e2ea689f5c54a/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/0aa644ac2cc09d88095e1c651bd8132aaa195aacf96d3ab27867f06c731bfeca/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/0a6ff205dfc4662f7ca8a1d585f47e4f1a79e0812cf247441df7f32571e11347/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-428:{mountpoint:/var/lib/containers/storage/overlay/c4f984daea71855ce715abad66ab6fc58c007a6db146527f636a2eac3d86a171/merged major:0 minor:428 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/2dea0f763806f836c092b8d251e11c0c4326dbf9b0656dc45c1d305e0d945a4d/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/d4bf2e42cd79dd4d93fb4ee6a61ae785a84ca8ba88b981459fe096c5ad9ba5b0/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-434:{mountpoint:/var/lib/containers/storage/overlay/dd8c5fcc82db04c5ac4face893a65f98a744958e0193874bfbd07fecd8fa3dfe/merged major:0 minor:434 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/d9a464bd8376e9eb20226727424e3c2762130e7c71504cd25e9d3719819cee4b/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/f46fffc434ff174b4c8fea89e8673080cb7ff2e2d7380662a1e7f54be2eec094/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-447:{mountpoint:/var/lib/containers/storage/overlay/de5a62bb0687e34d0b4cd36bc969122b032ae8755769e16cfce6aa660bc568b3/merged major:0 minor:447 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/70836e731497d09f5bf22485dd0df6decf82f4e7e809205effcfb55eeb776a64/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/02fd9586c04cf0b4b4164c10e8a191d0ef5393f2c4036fb10ed1fa4b485566c3/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/3be66c89ba6eeac962764dbf2751388958125b2bfa164e93c2573b810cb740fb/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/c1fd5ec8d6eda374a3b9168f607272ce7c9940c3d7747b100b92e3a9d84980ca/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/c52c5d0c889f8a5a73dbc99bc102d399e531583f64c94f61a079a6bd1a7b2df6/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/22701d0390e58ee05129f9b0e0f268d08d0353f81608487664903ff3eb5c6ab1/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/2a486fb5e3d7f639da033086abe2252d72e064698a0a0a4f4dde53b36c08b56b/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/05c421fc0a8c0612c5abe7d9fc4c7a0be28b34a59378e2d06ab1f759d760e83f/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/a6c23d13e197a8b77339f52da1936a375f33f9b2eb984dc000a614fe9325048b/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/a3c979dc9e7a7bfdf86ae3858f93e5f3b626c2e91078d0296371278cbd380758/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 01:20:44.063380 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.059882    5650 manager.go:217] Machine: {Timestamp:2026-10-31 01:20:44.05938456 +0000 UTC m=+0.063098841 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3093431 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2082908ab17fa0d0a29ca0b909c15f SystemUUID:ec208290-8ab1-7fa0-d0a2-9ca0b909c15f BootID:5f1a19c8-7324-48a2-8b26-c1e71927a23d Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/02c05e0c-b77a-45e4-b49b-1bed7e7588d0/volumes/kubernetes.io~projected/kube-api-access-v74ww DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:240 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de2a3af0-1220-4a82-a77a-772f99694285/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:263 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:377 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2a7902ff147c883af26f277241b0fa036f5e800913f8d7aeb1709878f1d15002/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5da57e3fdb12e9c5a9f6d66de998f92b009e424cd0f4f55a19e8644e439d3c93/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:283 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:283 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5546da65d949c1745ed589223b92c19947df5bf8b7946d4d320a55cb5d8e2836/userdata/shm DeviceMajor:0 DeviceMinor:423 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f62e7b940902c55bf18ce7cbe90f35be40ea9216ce7ade5ee556fcac904d1e55/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/76b7d8bd-5205-4243-a45f-c208a65bbc30/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:294 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/86947040-78ac-43d5-b932-9acd63cf43e7/volumes/kubernetes.io~projected/kube-api-access-v9jkq DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b0cbe408-97f0-48da-ab7d-ab4d144f0796/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-158 DeviceMajor:0 DeviceMinor:158 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76b7d8bd-5205-4243-a45f-c208a65bbc30/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:293 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-434 DeviceMajor:0 DeviceMinor:434 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/52e25fa9-6c04-453a-a0b2-cca32a428a5a/volumes/kubernetes.io~projected/kube-api-access-nw5bx DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1bc364fd433ed3d2f63c9112e2d37ed896872d6d972a9b39d31c407825d5bd70/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ce7c59f0-b861-41ab-a5ec-4f37bba8a619/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:275 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/76b7d8bd-5205-4243-a45f-c208a65bbc30/volumes/kubernetes.io~projected/kube-api-access-fdzbp DeviceMajor:0 DeviceMinor:295 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/60e7e4bc-764b-49e0-bb92-dea358fafcc5/volumes/kubernetes.io~projected/kube-api-access-9c8rn DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:380 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0cbe408-97f0-48da-ab7d-ab4d144f0796/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:277 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/76b7d8bd-5205-4243-a45f-c208a65bbc30/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:288 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:376 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/de2a3af0-1220-4a82-a77a-772f99694285/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:259 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:278 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:279 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:382 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:374 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7c02c9268fafa233a2d4682eb8d256e914786e9ccc7d60545d7e35e6ef4ff15/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:276 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:279 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5756be9b2413eb4946f209442f92e79b3230e1fddd09d95ce3c77a7d5202e9d2/userdata/shm DeviceMajor:0 DeviceMinor:384 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce7c59f0-b861-41ab-a5ec-4f37bba8a619/volumes/kubernetes.io~projected/kube-api-access-7pdph DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:378 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:367 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f8449422-f48d-4df2-8548-9de1c5339fd0/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:420 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/60e7e4bc-764b-49e0-bb92-dea358fafcc5/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:282 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~projected/kube-api-access-qt84f DeviceMajor:0 DeviceMinor:284 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/81c10625-0368-49c7-89cf-eb51f06005d0/volumes/kubernetes.io~projected/kube-api-access-v8tn8 DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5d3ecbcd60a975ff2c124eb66e9c9f22998bce80d5d69ad3359575bf6360e19/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:379 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce3225bd66cc30585aa3e235c6f2e0b285bbbcc7e2d207284a3ea74582082a10/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9e6beef-017b-49a6-81a3-45b8faf145fc/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:416 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-447 DeviceMajor:0 DeviceMinor:447 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6b7841e1-63bb-4e71-972e-1020c47d2ccb/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b0cbe408-97f0-48da-ab7d-ab4d144f0796/volumes/kubernetes.io~projected/kube-api-access-xmwzw DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de2a3af0-1220-4a82-a77a-772f99694285/volumes/kubernetes.io~projected/kube-api-access-qzjfj DeviceMajor:0 DeviceMinor:264 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/801d38c689ae4e6ea6e4a69f864ddc10453c1012b70f6632a1dfe0a7496e7659/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:367 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f2a0313973840437025dbb2eb884792953444175adb06ecbcb4f23a435c91a6/userdata/shm DeviceMajor:0 DeviceMinor:267 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:279 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c/volumes/kubernetes.io~projected/kube-api-access-9xs45 DeviceMajor:0 DeviceMinor:421 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-428 DeviceMajor:0 DeviceMinor:428 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243/volumes/kubernetes.io~projected/kube-api-access-n4cbn DeviceMajor:0 DeviceMinor:241 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16f972cb52728637abb62daf07e0ef221730ee2f44b1d9174bba586642ea3548/userdata/shm DeviceMajor:0 DeviceMinor:426 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6b7841e1-63bb-4e71-972e-1020c47d2ccb/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/aa56c5f6-4784-4b24-9daa-d17f7b296d8b/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a0b77182a4baaaa91fe5f386a9cf4c77cb619179e196a9f7fdd07b82287aae76/userdata/shm DeviceMajor:0 DeviceMinor:285 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:375 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0fd8f4b05e04837e0e0b26e5608e81495837e6095d72ec6fad2bef34dcaf7974/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76b7d8bd-5205-4243-a45f-c208a65bbc30/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:292 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d9e6beef-017b-49a6-81a3-45b8faf145fc/volumes/kubernetes.io~projected/kube-api-access-5md27 DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6b7841e1-63bb-4e71-972e-1020c47d2ccb/volumes/kubernetes.io~projected/kube-api-access-kh6l2 DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/197e8bd8725348bb6a7c30199d16b76d356a4e5051055216100b630d6bb51ae3/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:366 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~projected/kube-api-access-gzk26 DeviceMajor:0 DeviceMinor:383 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef2c4ca52dc80676d5f985fb9764888ef18effe4257bce45edfb81a5923c5653/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:280 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/76b7d8bd-5205-4243-a45f-c208a65bbc30/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:291 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cee0eb429cb99610539937858f608b35ded58d6f2ac2b56d040f8bba7b7b9916/userdata/shm DeviceMajor:0 DeviceMinor:422 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/167a4eb856c92f9ac3fc51b87c43d6bca9a72a347aae3bf431c7b8203a9de001/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/db94c614-bfe5-4328-b1aa-770694e4fca1/volumes/kubernetes.io~projected/kube-api-access-xwd6x DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:381 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f82d934690d4d27dd037f936a6e4c17907e23379df72d3eb8ba44e6d2c6d442/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eeb3287803cb703ffb46992869dd163fed0f1073887d3f04f9c2ae218331521e/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:279 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa56c5f6-4784-4b24-9daa-d17f7b296d8b/volumes/kubernetes.io~projected/kube-api-access-7g8zb DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:281 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ec3e19fc-4890-424b-86ef-f5ad1e926158/volumes/kubernetes.io~projected/kube-api-access-qr95h DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9849baee4ffe03763bf3fa9a1bc91f2b9d3a2c828d6cb0d18680c50d8505efa7/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:367 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:236 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/76b7d8bd-5205-4243-a45f-c208a65bbc30/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:287 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8aa9f886-74c9-442f-89fc-beb5fc0bf260/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:367 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:16f972cb5272863 MacAddress:8e:ef:91:00:cd:8e Speed:10000 Mtu:8901} {Name:197e8bd8725348b MacAddress:96:95:7c:4f:1e:a9 Speed:10000 Mtu:8901} {Name:5546da65d949c17 MacAddress:f2:2a:5d:e2:78:1f Speed:10000 Mtu:8901} {Name:5756be9b2413eb4 MacAddress:f6:9c:69:62:2f:4c Speed:10000 Mtu:8901} {Name:a0b77182a4baaaa MacAddress:e6:86:ff:ef:ba:36 Speed:10000 Mtu:8901} {Name:b7c02c9268fafa2 MacAddress:ea:f6:e2:2a:e4:11 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:dd:cc:0e:59 Speed:0 Mtu:9001} {Name:br-int MacAddress:86:2e:dd:54:42:16 Speed:0 Mtu:8901} {Name:ce3225bd66cc305 MacAddress:6a:32:0a:35:b5:44 Speed:10000 Mtu:8901} {Name:cee0eb429cb9961 MacAddress:de:c4:3f:d9:03:3f Speed:10000 Mtu:8901} {Name:eeb3287803cb703 MacAddress:3e:e2:75:f1:bf:5f Speed:10000 Mtu:8901} {Name:ef2c4ca52dc8067 MacAddress:ce:9c:6c:3c:18:bc Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:dd:cc:0e:59 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:8e:8f:ec:04:8c:cb Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:1a:81:46:b7:56:69 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 01:20:44.063380 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.060283    5650 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 01:20:44.063380 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.060329    5650 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 01:20:44.063380 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.060707    5650 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 01:20:44.063380 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.060732    5650 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-98.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 01:20:44.063380 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.060950    5650 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 01:20:44.063380 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.060963    5650 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 01:20:44.063380 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.060982    5650 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 01:20:44.063380 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.061002    5650 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 01:20:44.063380 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.061127    5650 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 01:20:44.063380 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.061366    5650 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 01:20:44.063380 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.061538    5650 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 01:20:44.063380 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.061583    5650 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 01:20:44.063380 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.061598    5650 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 01:20:44.063380 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.061613    5650 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 01:20:44.063380 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.061626    5650 kubelet.go:405] "Adding apiserver pod source"
Oct 31 01:20:44.063380 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.061638    5650 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 01:20:44.063380 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.062996    5650 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 01:20:44.063380 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.063011    5650 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 01:20:44.064367 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.063611    5650 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 01:20:44.067515 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.067496    5650 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 01:20:44.067515 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.067512    5650 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 01:20:44.067645 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.067618    5650 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 01:20:44.067816 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.067805    5650 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 01:20:44.067846 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.067835    5650 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 01:20:44.067935 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.067925    5650 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 01:20:44.067970 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.067945    5650 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 01:20:44.067970 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.067957    5650 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 01:20:44.067970 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.067966    5650 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 01:20:44.068055 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.067972    5650 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 01:20:44.068055 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.067978    5650 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 01:20:44.068055 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.067985    5650 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 01:20:44.068055 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.067991    5650 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 01:20:44.068055 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.067998    5650 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 01:20:44.068055 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.068004    5650 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 01:20:44.068055 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.068015    5650 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 01:20:44.068055 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.068027    5650 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 01:20:44.068055 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.068055    5650 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 01:20:44.068326 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.068064    5650 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 01:20:44.071415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.071402    5650 server.go:1263] "Started kubelet"
Oct 31 01:20:44.071831 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.071474    5650 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 01:20:44.071831 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.071540    5650 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 01:20:44.071831 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.071610    5650 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 01:20:44.071963 ip-10-0-133-98 systemd[1]: Started Kubernetes Kubelet.
Oct 31 01:20:44.072093 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.071995    5650 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 01:20:44.073285 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.073267    5650 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 01:20:44.077325 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.076134    5650 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 01:20:44.077325 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.076151    5650 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 01:20:44.077325 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.076271    5650 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 01:20:44.077325 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.076283    5650 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 01:20:44.077325 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.076303    5650 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 01:14:34 +0000 UTC" deadline="2028-05-07 20:11:06.646876544 +0000 UTC"
Oct 31 01:20:44.077325 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.076339    5650 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13314h50m22.570541264s"
Oct 31 01:20:44.077325 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.076805    5650 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 01:20:44.077325 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:20:44.077116    5650 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-98.us-east-2.compute.internal\" not found"
Oct 31 01:20:44.081538 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.079475    5650 factory.go:55] Registering systemd factory
Oct 31 01:20:44.081538 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.079499    5650 factory.go:223] Registration of the systemd container factory successfully
Oct 31 01:20:44.081538 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.080669    5650 factory.go:153] Registering CRI-O factory
Oct 31 01:20:44.081538 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.080693    5650 factory.go:223] Registration of the crio container factory successfully
Oct 31 01:20:44.081538 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.080943    5650 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 01:20:44.081538 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.080970    5650 factory.go:103] Registering Raw factory
Oct 31 01:20:44.081538 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.080989    5650 manager.go:1199] Started watching for new ooms in manager
Oct 31 01:20:44.081538 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.081515    5650 manager.go:319] Starting recovery of all containers
Oct 31 01:20:44.085411 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085370    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80" volumeName="kubernetes.io/configmap/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-metrics-client-ca" seLinuxMountContext=""
Oct 31 01:20:44.085540 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085414    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52e25fa9-6c04-453a-a0b2-cca32a428a5a" volumeName="kubernetes.io/configmap/52e25fa9-6c04-453a-a0b2-cca32a428a5a-serviceca" seLinuxMountContext=""
Oct 31 01:20:44.085540 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085429    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76b7d8bd-5205-4243-a45f-c208a65bbc30" volumeName="kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Oct 31 01:20:44.085540 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085462    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce7c59f0-b861-41ab-a5ec-4f37bba8a619" volumeName="kubernetes.io/configmap/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 31 01:20:44.085540 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085501    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de2a3af0-1220-4a82-a77a-772f99694285" volumeName="kubernetes.io/configmap/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 31 01:20:44.085540 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085514    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80" volumeName="kubernetes.io/empty-dir/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-config-out" seLinuxMountContext=""
Oct 31 01:20:44.085540 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085526    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76b7d8bd-5205-4243-a45f-c208a65bbc30" volumeName="kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-tls" seLinuxMountContext=""
Oct 31 01:20:44.085540 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085539    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa9f886-74c9-442f-89fc-beb5fc0bf260" volumeName="kubernetes.io/empty-dir/8aa9f886-74c9-442f-89fc-beb5fc0bf260-config-out" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085555    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de2a3af0-1220-4a82-a77a-772f99694285" volumeName="kubernetes.io/projected/de2a3af0-1220-4a82-a77a-772f99694285-kube-api-access-qzjfj" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085568    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de2a3af0-1220-4a82-a77a-772f99694285" volumeName="kubernetes.io/secret/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-tls" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085579    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec3e19fc-4890-424b-86ef-f5ad1e926158" volumeName="kubernetes.io/configmap/ec3e19fc-4890-424b-86ef-f5ad1e926158-cni-binary-copy" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085602    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80" volumeName="kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085613    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b7841e1-63bb-4e71-972e-1020c47d2ccb" volumeName="kubernetes.io/projected/6b7841e1-63bb-4e71-972e-1020c47d2ccb-kube-api-access-kh6l2" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085625    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b7841e1-63bb-4e71-972e-1020c47d2ccb" volumeName="kubernetes.io/secret/6b7841e1-63bb-4e71-972e-1020c47d2ccb-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085641    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa9f886-74c9-442f-89fc-beb5fc0bf260" volumeName="kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085653    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa9f886-74c9-442f-89fc-beb5fc0bf260" volumeName="kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085665    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa9f886-74c9-442f-89fc-beb5fc0bf260" volumeName="kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-grpc-tls" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085677    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243" volumeName="kubernetes.io/secret/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085690    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b7841e1-63bb-4e71-972e-1020c47d2ccb" volumeName="kubernetes.io/configmap/6b7841e1-63bb-4e71-972e-1020c47d2ccb-ovnkube-config" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085701    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa9f886-74c9-442f-89fc-beb5fc0bf260" volumeName="kubernetes.io/projected/8aa9f886-74c9-442f-89fc-beb5fc0bf260-tls-assets" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085710    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa9f886-74c9-442f-89fc-beb5fc0bf260" volumeName="kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-thanos-prometheus-http-client-file" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085742    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce7c59f0-b861-41ab-a5ec-4f37bba8a619" volumeName="kubernetes.io/projected/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-kube-api-access-7pdph" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085757    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f" volumeName="kubernetes.io/secret/308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f-agent-certs" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085771    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa9f886-74c9-442f-89fc-beb5fc0bf260" volumeName="kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-prometheus-k8s-tls" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085781    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa56c5f6-4784-4b24-9daa-d17f7b296d8b" volumeName="kubernetes.io/configmap/aa56c5f6-4784-4b24-9daa-d17f7b296d8b-config-volume" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085793    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76b7d8bd-5205-4243-a45f-c208a65bbc30" volumeName="kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085806    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa9f886-74c9-442f-89fc-beb5fc0bf260" volumeName="kubernetes.io/empty-dir/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-k8s-db" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085828    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa9f886-74c9-442f-89fc-beb5fc0bf260" volumeName="kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-config" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085841    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9e6beef-017b-49a6-81a3-45b8faf145fc" volumeName="kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085848    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db94c614-bfe5-4328-b1aa-770694e4fca1" volumeName="kubernetes.io/projected/db94c614-bfe5-4328-b1aa-770694e4fca1-kube-api-access-xwd6x" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085865    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de2a3af0-1220-4a82-a77a-772f99694285" volumeName="kubernetes.io/empty-dir/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-textfile" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085873    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b7841e1-63bb-4e71-972e-1020c47d2ccb" volumeName="kubernetes.io/configmap/6b7841e1-63bb-4e71-972e-1020c47d2ccb-env-overrides" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085881    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76b7d8bd-5205-4243-a45f-c208a65bbc30" volumeName="kubernetes.io/projected/76b7d8bd-5205-4243-a45f-c208a65bbc30-kube-api-access-fdzbp" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085890    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86947040-78ac-43d5-b932-9acd63cf43e7" volumeName="kubernetes.io/empty-dir/86947040-78ac-43d5-b932-9acd63cf43e7-tmp-dir" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085897    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa9f886-74c9-442f-89fc-beb5fc0bf260" volumeName="kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-metrics-client-certs" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085905    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0cbe408-97f0-48da-ab7d-ab4d144f0796" volumeName="kubernetes.io/projected/b0cbe408-97f0-48da-ab7d-ab4d144f0796-kube-api-access-xmwzw" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085913    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243" volumeName="kubernetes.io/configmap/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243-metrics-client-ca" seLinuxMountContext=""
Oct 31 01:20:44.085895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085920    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9e6beef-017b-49a6-81a3-45b8faf145fc" volumeName="kubernetes.io/projected/d9e6beef-017b-49a6-81a3-45b8faf145fc-kube-api-access-5md27" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085929    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa9f886-74c9-442f-89fc-beb5fc0bf260" volumeName="kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-configmap-metrics-client-ca" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085938    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa9f886-74c9-442f-89fc-beb5fc0bf260" volumeName="kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085946    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa9f886-74c9-442f-89fc-beb5fc0bf260" volumeName="kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-kube-rbac-proxy" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085954    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0cbe408-97f0-48da-ab7d-ab4d144f0796" volumeName="kubernetes.io/empty-dir/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-tuned" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085961    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce7c59f0-b861-41ab-a5ec-4f37bba8a619" volumeName="kubernetes.io/secret/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085968    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2aa13b7e-bcfc-4ca6-96bd-086caf620c9c" volumeName="kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085976    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60e7e4bc-764b-49e0-bb92-dea358fafcc5" volumeName="kubernetes.io/secret/60e7e4bc-764b-49e0-bb92-dea358fafcc5-cert" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085983    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86947040-78ac-43d5-b932-9acd63cf43e7" volumeName="kubernetes.io/projected/86947040-78ac-43d5-b932-9acd63cf43e7-kube-api-access-v9jkq" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.085997    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa9f886-74c9-442f-89fc-beb5fc0bf260" volumeName="kubernetes.io/projected/8aa9f886-74c9-442f-89fc-beb5fc0bf260-kube-api-access-gzk26" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086005    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa56c5f6-4784-4b24-9daa-d17f7b296d8b" volumeName="kubernetes.io/secret/aa56c5f6-4784-4b24-9daa-d17f7b296d8b-metrics-tls" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086013    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec3e19fc-4890-424b-86ef-f5ad1e926158" volumeName="kubernetes.io/configmap/ec3e19fc-4890-424b-86ef-f5ad1e926158-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086021    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80" volumeName="kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-web-config" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086029    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80" volumeName="kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-secret-alertmanager-main-tls" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086037    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52e25fa9-6c04-453a-a0b2-cca32a428a5a" volumeName="kubernetes.io/projected/52e25fa9-6c04-453a-a0b2-cca32a428a5a-kube-api-access-nw5bx" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086044    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76b7d8bd-5205-4243-a45f-c208a65bbc30" volumeName="kubernetes.io/configmap/76b7d8bd-5205-4243-a45f-c208a65bbc30-metrics-client-ca" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086052    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa9f886-74c9-442f-89fc-beb5fc0bf260" volumeName="kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086059    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa9f886-74c9-442f-89fc-beb5fc0bf260" volumeName="kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086067    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce7c59f0-b861-41ab-a5ec-4f37bba8a619" volumeName="kubernetes.io/empty-dir/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-data-volume" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086082    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80" volumeName="kubernetes.io/configmap/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086090    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80" volumeName="kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086098    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60e7e4bc-764b-49e0-bb92-dea358fafcc5" volumeName="kubernetes.io/projected/60e7e4bc-764b-49e0-bb92-dea358fafcc5-kube-api-access-9c8rn" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086108    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76b7d8bd-5205-4243-a45f-c208a65bbc30" volumeName="kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086121    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81c10625-0368-49c7-89cf-eb51f06005d0" volumeName="kubernetes.io/configmap/81c10625-0368-49c7-89cf-eb51f06005d0-cni-binary-copy" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086133    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81c10625-0368-49c7-89cf-eb51f06005d0" volumeName="kubernetes.io/projected/81c10625-0368-49c7-89cf-eb51f06005d0-kube-api-access-v8tn8" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086146    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa9f886-74c9-442f-89fc-beb5fc0bf260" volumeName="kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086158    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de2a3af0-1220-4a82-a77a-772f99694285" volumeName="kubernetes.io/secret/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086171    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80" volumeName="kubernetes.io/empty-dir/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-alertmanager-main-db" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086183    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b7841e1-63bb-4e71-972e-1020c47d2ccb" volumeName="kubernetes.io/configmap/6b7841e1-63bb-4e71-972e-1020c47d2ccb-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086195    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81c10625-0368-49c7-89cf-eb51f06005d0" volumeName="kubernetes.io/configmap/81c10625-0368-49c7-89cf-eb51f06005d0-multus-daemon-config" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086206    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa9f886-74c9-442f-89fc-beb5fc0bf260" volumeName="kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086219    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa9f886-74c9-442f-89fc-beb5fc0bf260" volumeName="kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086234    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db94c614-bfe5-4328-b1aa-770694e4fca1" volumeName="kubernetes.io/configmap/db94c614-bfe5-4328-b1aa-770694e4fca1-iptables-alerter-script" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086254    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de2a3af0-1220-4a82-a77a-772f99694285" volumeName="kubernetes.io/configmap/de2a3af0-1220-4a82-a77a-772f99694285-metrics-client-ca" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086275    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80" volumeName="kubernetes.io/projected/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-tls-assets" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086293    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80" volumeName="kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-cluster-tls-config" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086314    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80" volumeName="kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086340    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76b7d8bd-5205-4243-a45f-c208a65bbc30" volumeName="kubernetes.io/empty-dir/76b7d8bd-5205-4243-a45f-c208a65bbc30-tmp" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086359    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76b7d8bd-5205-4243-a45f-c208a65bbc30" volumeName="kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086374    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0cbe408-97f0-48da-ab7d-ab4d144f0796" volumeName="kubernetes.io/empty-dir/b0cbe408-97f0-48da-ab7d-ab4d144f0796-tmp" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086391    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243" volumeName="kubernetes.io/projected/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243-kube-api-access-n4cbn" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086406    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec3e19fc-4890-424b-86ef-f5ad1e926158" volumeName="kubernetes.io/configmap/ec3e19fc-4890-424b-86ef-f5ad1e926158-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086430    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f" volumeName="kubernetes.io/configmap/308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f-konnectivity-ca" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086467    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76b7d8bd-5205-4243-a45f-c208a65bbc30" volumeName="kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-grpc-tls" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086485    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa9f886-74c9-442f-89fc-beb5fc0bf260" volumeName="kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-web-config" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086501    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec3e19fc-4890-424b-86ef-f5ad1e926158" volumeName="kubernetes.io/projected/ec3e19fc-4890-424b-86ef-f5ad1e926158-kube-api-access-qr95h" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086518    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02c05e0c-b77a-45e4-b49b-1bed7e7588d0" volumeName="kubernetes.io/projected/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-kube-api-access-v74ww" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086534    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa56c5f6-4784-4b24-9daa-d17f7b296d8b" volumeName="kubernetes.io/empty-dir/aa56c5f6-4784-4b24-9daa-d17f7b296d8b-tmp-dir" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086549    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243" volumeName="kubernetes.io/secret/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243-prometheus-operator-tls" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086581    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80" volumeName="kubernetes.io/projected/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-kube-api-access-qt84f" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086599    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80" volumeName="kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-config-volume" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086616    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa56c5f6-4784-4b24-9daa-d17f7b296d8b" volumeName="kubernetes.io/projected/aa56c5f6-4784-4b24-9daa-d17f7b296d8b-kube-api-access-7g8zb" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086631    5650 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8449422-f48d-4df2-8548-9de1c5339fd0" volumeName="kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret" seLinuxMountContext=""
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086647    5650 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 01:20:44.087313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.086658    5650 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 01:20:44.093907 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.093856    5650 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:20:44.094189 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.094167    5650 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:20:44.098411 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.098384    5650 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:20:44.098672 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.098657    5650 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 01:20:44.108410 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.108374    5650 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 01:20:44.110043 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.109872    5650 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 01:20:44.110043 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.109893    5650 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 01:20:44.110043 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.109915    5650 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 01:20:44.110043 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:20:44.109953    5650 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 01:20:44.114963 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.114936    5650 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:20:44.122895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.122862    5650 generic.go:356] "Generic (PLEG): container finished" podID="ec3e19fc-4890-424b-86ef-f5ad1e926158" containerID="c29de0b235c3d20d87f4a0cec521f7fb1a0524eca5b5d527fefc142ec4e6d175" exitCode=0
Oct 31 01:20:44.122995 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.122897    5650 generic.go:356] "Generic (PLEG): container finished" podID="ec3e19fc-4890-424b-86ef-f5ad1e926158" containerID="25c8e34977842b7c816a16c4e4032a4c099da48e0ed22fe680b12ded1b4ff90c" exitCode=0
Oct 31 01:20:44.122995 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.122908    5650 generic.go:356] "Generic (PLEG): container finished" podID="ec3e19fc-4890-424b-86ef-f5ad1e926158" containerID="a034c68cb6dc0afb42e3348e3ad8c60819824cf0eb95a3b1114fdc1fcc8c247a" exitCode=0
Oct 31 01:20:44.122995 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.122921    5650 generic.go:356] "Generic (PLEG): container finished" podID="ec3e19fc-4890-424b-86ef-f5ad1e926158" containerID="5506323c3914d48883f1e18256a04bf102a41a698f26fc706a8e4b288faab1eb" exitCode=0
Oct 31 01:20:44.122995 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.122934    5650 generic.go:356] "Generic (PLEG): container finished" podID="ec3e19fc-4890-424b-86ef-f5ad1e926158" containerID="ef579280d5803ddba0a67dc0e4fe5ae9e485903371cc26be617f979a841bd747" exitCode=0
Oct 31 01:20:44.122995 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.122945    5650 generic.go:356] "Generic (PLEG): container finished" podID="ec3e19fc-4890-424b-86ef-f5ad1e926158" containerID="ab40995ed15e2a02727e79ce17c0bccd74406086035b7c4a11d7a04bbbcc8561" exitCode=0
Oct 31 01:20:44.124839 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.124812    5650 generic.go:356] "Generic (PLEG): container finished" podID="a2a6228f7ebdca678429d632d3fb70c4" containerID="1df7e0014ee5a640c7d1d3465f00a2f672d0cf2e801516c8ee9152a472a69da8" exitCode=0
Oct 31 01:20:44.130900 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.130872    5650 generic.go:356] "Generic (PLEG): container finished" podID="de2a3af0-1220-4a82-a77a-772f99694285" containerID="d8ef985f4efcdbcab7da1dd49209ef13764d3dfc281b3b45dfd65838f98926b9" exitCode=0
Oct 31 01:20:44.142316 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.142288    5650 generic.go:356] "Generic (PLEG): container finished" podID="8aa9f886-74c9-442f-89fc-beb5fc0bf260" containerID="2eaf6d057afdd3fa6206f6a74542d19c3d2e45e4d4ecafc5ebf8e544583d3b79" exitCode=0
Oct 31 01:20:44.149224 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.149200    5650 generic.go:356] "Generic (PLEG): container finished" podID="0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80" containerID="a7a3fd5806b8c7341e03e5d8e00b14a4c9a08c6f585024689294b8f1c8d663e2" exitCode=0
Oct 31 01:20:44.193187 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.193111    5650 manager.go:324] Recovery completed
Oct 31 01:20:44.210276 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:20:44.210245    5650 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 31 01:20:44.216412 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.216387    5650 cpu_manager.go:226] "Starting" policy="none"
Oct 31 01:20:44.216412 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.216406    5650 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 01:20:44.216586 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.216429    5650 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 01:20:44.216636 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.216610    5650 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 01:20:44.216671 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.216627    5650 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 01:20:44.216671 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.216651    5650 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 01:20:44.216671 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.216659    5650 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 01:20:44.216671 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.216667    5650 policy_none.go:50] "Start"
Oct 31 01:20:44.216773 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.216680    5650 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 01:20:44.216773 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.216693    5650 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 01:20:44.216836 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.216825    5650 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 01:20:44.216869 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.216836    5650 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 01:20:44.216869 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.216842    5650 policy_none.go:44] "Start"
Oct 31 01:20:44.221525 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.221494    5650 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 01:20:44.224090 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.224076    5650 manager.go:341] "Starting Device Plugin manager"
Oct 31 01:20:44.224589 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:20:44.224111    5650 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 01:20:44.224589 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.224590    5650 server.go:87] "Starting device plugin registration server"
Oct 31 01:20:44.224869 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.224858    5650 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 01:20:44.224922 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.224870    5650 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 01:20:44.224968 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.224951    5650 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 01:20:44.225073 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.225044    5650 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 01:20:44.225199 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.225080    5650 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 01:20:44.225199 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.225092    5650 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 01:20:44.225749 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.225178    5650 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T01:20:44.225067919Z","UUID":"603a7de9-6337-4cb2-aaeb-b035e95042f9","Handler":null,"Name":"","Endpoint":""}
Oct 31 01:20:44.227585 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:20:44.227495    5650 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 01:20:44.227585 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.227571    5650 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 01:20:44.227733 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.227633    5650 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 01:20:44.325239 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.325202    5650 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:20:44.328142 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.328120    5650 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-98.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:20:44.328271 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.328154    5650 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-98.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:20:44.328271 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.328165    5650 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-98.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:20:44.328271 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.328209    5650 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:20:44.342244 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.342215    5650 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:20:44.342370 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.342292    5650 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:20:44.410893 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.410825    5650 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal"]
Oct 31 01:20:44.411590 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.411539    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal" event={"ID":"a2a6228f7ebdca678429d632d3fb70c4","Type":"ContainerStarted","Data":"6fb46a21ef135cc64da83a5afcb4a0a0259645f1684dd7e48140dda332177113"}
Oct 31 01:20:44.411673 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.411597    5650 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:20:44.411673 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.411614    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal" event={"ID":"a2a6228f7ebdca678429d632d3fb70c4","Type":"ContainerDied","Data":"1df7e0014ee5a640c7d1d3465f00a2f672d0cf2e801516c8ee9152a472a69da8"}
Oct 31 01:20:44.411673 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.411623    5650 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:20:44.411673 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.411637    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal" event={"ID":"a2a6228f7ebdca678429d632d3fb70c4","Type":"ContainerStarted","Data":"4f82d934690d4d27dd037f936a6e4c17907e23379df72d3eb8ba44e6d2c6d442"}
Oct 31 01:20:44.411798 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.411679    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal" event={"ID":"7b4242bdffafd37f633de48cdce1cc09","Type":"ContainerStarted","Data":"6eae2816c9f8ae6f26610a9d26d7a5ac924261135af2d8f4a979b68562100e8c"}
Oct 31 01:20:44.411798 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.411698    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal" event={"ID":"7b4242bdffafd37f633de48cdce1cc09","Type":"ContainerStarted","Data":"167a4eb856c92f9ac3fc51b87c43d6bca9a72a347aae3bf431c7b8203a9de001"}
Oct 31 01:20:44.424731 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.424700    5650 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:20:44.424836 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:20:44.424757    5650 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:20:44.425097 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.425080    5650 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:20:44.425135 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:20:44.425110    5650 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:20:44.500496 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.500442    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7b4242bdffafd37f633de48cdce1cc09-config\") pod \"kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal\" (UID: \"7b4242bdffafd37f633de48cdce1cc09\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:20:44.500496 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.500500    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a2a6228f7ebdca678429d632d3fb70c4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal\" (UID: \"a2a6228f7ebdca678429d632d3fb70c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:20:44.500718 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.500538    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2a6228f7ebdca678429d632d3fb70c4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal\" (UID: \"a2a6228f7ebdca678429d632d3fb70c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:20:44.601126 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.601083    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a2a6228f7ebdca678429d632d3fb70c4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal\" (UID: \"a2a6228f7ebdca678429d632d3fb70c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:20:44.601126 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.601129    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2a6228f7ebdca678429d632d3fb70c4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal\" (UID: \"a2a6228f7ebdca678429d632d3fb70c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:20:44.601319 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.601149    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7b4242bdffafd37f633de48cdce1cc09-config\") pod \"kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal\" (UID: \"7b4242bdffafd37f633de48cdce1cc09\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:20:44.601319 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.601193    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7b4242bdffafd37f633de48cdce1cc09-config\") pod \"kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal\" (UID: \"7b4242bdffafd37f633de48cdce1cc09\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:20:44.601319 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.601204    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a2a6228f7ebdca678429d632d3fb70c4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal\" (UID: \"a2a6228f7ebdca678429d632d3fb70c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:20:44.601319 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:44.601204    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2a6228f7ebdca678429d632d3fb70c4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal\" (UID: \"a2a6228f7ebdca678429d632d3fb70c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:20:45.062718 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.062679    5650 apiserver.go:52] "Watching apiserver"
Oct 31 01:20:45.086095 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.086065    5650 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 01:20:45.086733 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.086701    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-v9wzp","kube-system/kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-vfk9h","openshift-cluster-node-tuning-operator/tuned-m7xdt","openshift-dns/dns-default-kmxk8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal","openshift-multus/multus-fqpcx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv","openshift-monitoring/thanos-querier-86d478dd89-l66f7","openshift-network-diagnostics/network-check-target-jn7dx","openshift-insights/insights-runtime-extractor-8n2r2","openshift-network-operator/iptables-alerter-lhhnx","kube-system/konnectivity-agent-wnmfp","openshift-monitoring/alertmanager-main-0","openshift-dns/node-resolver-xghmx","openshift-image-registry/node-ca-tzz5q","openshift-ovn-kubernetes/ovnkube-node-tmh79","openshift-monitoring/prometheus-operator-9cd89b74c-97j2h","openshift-monitoring/node-exporter-cdkr4","openshift-multus/multus-additional-cni-plugins-ll9gg","openshift-monitoring/prometheus-k8s-0","kube-system/global-pull-secret-syncer-5crfw"]
Oct 31 01:20:45.095580 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.095527    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 01:20:45.095580 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.095541    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 01:20:45.095580 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.095541    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 01:20:45.097893 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.097868    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 01:20:45.097893 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.097890    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 01:20:45.097893 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.097895    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 01:20:45.097893 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.097908    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gv7wl\""
Oct 31 01:20:45.098122 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.097927    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lbv59\""
Oct 31 01:20:45.098122 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.097944    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vjbgx\""
Oct 31 01:20:45.098122 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.097958    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-z69qs\""
Oct 31 01:20:45.098631 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.098611    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 01:20:45.098709 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.098635    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 01:20:45.102931 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.102910    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h7598\""
Oct 31 01:20:45.107144 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.107119    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 31 01:20:45.107232 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.107150    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 01:20:45.107338 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.107301    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 31 01:20:45.107436 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.107361    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 01:20:45.107436 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.107361    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 01:20:45.108487 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.108470    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 31 01:20:45.108566 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.108485    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 01:20:45.108566 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.108500    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-748gw\""
Oct 31 01:20:45.109622 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.109608    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 01:20:45.109622 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.109611    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 31 01:20:45.109738 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.109640    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 01:20:45.109907 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.109891    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 31 01:20:45.109972 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.109920    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 31 01:20:45.109972 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.109943    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bztmp\""
Oct 31 01:20:45.110048 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.109960    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 01:20:45.110048 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.109923    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-44nrr\""
Oct 31 01:20:45.110048 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.109974    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 01:20:45.112041 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.112020    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 31 01:20:45.112265 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.112043    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 01:20:45.112265 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.112056    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 01:20:45.112265 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.112079    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 01:20:45.112265 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.112085    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-83h262jp6g4hf\""
Oct 31 01:20:45.112265 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.112085    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 01:20:45.114214 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.114192    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vj55m\""
Oct 31 01:20:45.114214 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.114217    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 01:20:45.114347 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.114250    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d5hw4\""
Oct 31 01:20:45.114524 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.114507    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-cg6pk\""
Oct 31 01:20:45.114597 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.114512    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 31 01:20:45.114597 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.114542    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 01:20:45.114597 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.114550    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 31 01:20:45.114597 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.114547    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 31 01:20:45.114597 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.114569    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-svfbx\""
Oct 31 01:20:45.114597 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.114576    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 31 01:20:45.114837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.114586    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 31 01:20:45.114837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.114589    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 31 01:20:45.114837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.114714    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-am3j72sv96m6q\""
Oct 31 01:20:45.114837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.114744    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-j7vz8\""
Oct 31 01:20:45.114837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.114752    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-79nk6\""
Oct 31 01:20:45.114837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.114767    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 31 01:20:45.114837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.114768    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fnh8f\""
Oct 31 01:20:45.114837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.114780    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-82mbc\""
Oct 31 01:20:45.114837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.114769    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hm2lp\""
Oct 31 01:20:45.114837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.114805    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-j25cw\""
Oct 31 01:20:45.115171 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.115151    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 31 01:20:45.116327 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.116305    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-whdn9\""
Oct 31 01:20:45.116327 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.116318    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tsbmr\""
Oct 31 01:20:45.116441 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.116323    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 31 01:20:45.171883 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.171857    5650 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:20:45.172013 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.171990    5650 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:20:45.181869 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.181843    5650 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 01:20:45.185156 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.185134    5650 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:20:45.185262 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:20:45.185177    5650 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:20:45.185948 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.185935    5650 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:20:45.186013 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:20:45.185966    5650 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-98.us-east-2.compute.internal"
Oct 31 01:20:45.194411 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.194389    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:20:45.194525 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.194488    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 01:20:45.194525 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.194491    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 01:20:45.194609 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.194538    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 01:20:45.194609 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.194591    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 01:20:45.194743 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.194726    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 01:20:45.194805 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.194766    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 01:20:45.194901 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.194887    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 01:20:45.195285 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.195265    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 01:20:45.195285 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.195277    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 01:20:45.195419 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.195284    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 01:20:45.195419 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.195285    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 01:20:45.195419 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.195325    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 01:20:45.195419 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.195335    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 01:20:45.195419 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.195279    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 01:20:45.195419 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.195391    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 31 01:20:45.195419 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.195420    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 01:20:45.196526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.196509    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 31 01:20:45.196583 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.196532    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 31 01:20:45.197165 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.197145    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 01:20:45.197165 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.197160    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 01:20:45.197279 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.197183    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 01:20:45.197279 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.197203    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 01:20:45.197426 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.197414    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 01:20:45.197490 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.197477    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 01:20:45.197539 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.197496    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 01:20:45.197578 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.197543    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 01:20:45.197578 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.197546    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 01:20:45.197578 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.197563    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 01:20:45.198751 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.198735    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:20:45.198751 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.198745    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 01:20:45.199541 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.199514    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 01:20:45.201724 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.201702    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 01:20:45.201802 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.201726    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 01:20:45.201802 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.201711    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 01:20:45.203164 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203147    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8aa9f886-74c9-442f-89fc-beb5fc0bf260-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.203217 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203174    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.203217 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203191    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.203303 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203213    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.203303 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203241    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f-agent-certs\") pod \"konnectivity-agent-wnmfp\" (UID: \"308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f\") " pod="kube-system/konnectivity-agent-wnmfp"
Oct 31 01:20:45.203404 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203359    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.203404 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203395    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de2a3af0-1220-4a82-a77a-772f99694285-sys\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:45.203526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203404    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.203526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203419    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8aa9f886-74c9-442f-89fc-beb5fc0bf260-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.203526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203429    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81c10625-0368-49c7-89cf-eb51f06005d0-cni-binary-copy\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.203526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203481    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.203526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203501    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f8449422-f48d-4df2-8548-9de1c5339fd0-kubelet-config\") pod \"global-pull-secret-syncer-5crfw\" (UID: \"f8449422-f48d-4df2-8548-9de1c5339fd0\") " pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:20:45.203526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203524    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f-agent-certs\") pod \"konnectivity-agent-wnmfp\" (UID: \"308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f\") " pod="kube-system/konnectivity-agent-wnmfp"
Oct 31 01:20:45.203794 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203530    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.203794 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203537    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ec3e19fc-4890-424b-86ef-f5ad1e926158-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:20:45.203794 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203565    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-config\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.203794 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203587    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-tls\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:45.203794 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203613    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-config-volume\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.203794 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203656    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81c10625-0368-49c7-89cf-eb51f06005d0-cni-binary-copy\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.203794 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203707    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-web-config\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.203794 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203735    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db94c614-bfe5-4328-b1aa-770694e4fca1-iptables-alerter-script\") pod \"iptables-alerter-lhhnx\" (UID: \"db94c614-bfe5-4328-b1aa-770694e4fca1\") " pod="openshift-network-operator/iptables-alerter-lhhnx"
Oct 31 01:20:45.203794 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203778    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76b7d8bd-5205-4243-a45f-c208a65bbc30-tmp\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:45.204197 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203805    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ec3e19fc-4890-424b-86ef-f5ad1e926158-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:20:45.204197 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203831    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-config-volume\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.204197 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203835    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-tls\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:45.204197 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203808    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.204197 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203811    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-config\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.204197 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203878    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76b7d8bd-5205-4243-a45f-c208a65bbc30-tmp\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:45.204197 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203894    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.204197 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203899    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-web-config\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.204197 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203925    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-sysconfig\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.204197 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203950    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-web-config\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.204197 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203975    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db94c614-bfe5-4328-b1aa-770694e4fca1-iptables-alerter-script\") pod \"iptables-alerter-lhhnx\" (UID: \"db94c614-bfe5-4328-b1aa-770694e4fca1\") " pod="openshift-network-operator/iptables-alerter-lhhnx"
Oct 31 01:20:45.204197 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203995    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.204197 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.203987    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60e7e4bc-764b-49e0-bb92-dea358fafcc5-cert\") pod \"ingress-canary-v9wzp\" (UID: \"60e7e4bc-764b-49e0-bb92-dea358fafcc5\") " pod="openshift-ingress-canary/ingress-canary-v9wzp"
Oct 31 01:20:45.204197 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204059    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdzbp\" (UniqueName: \"kubernetes.io/projected/76b7d8bd-5205-4243-a45f-c208a65bbc30-kube-api-access-fdzbp\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:45.204197 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204066    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.204197 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204163    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.204197 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204166    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60e7e4bc-764b-49e0-bb92-dea358fafcc5-cert\") pod \"ingress-canary-v9wzp\" (UID: \"60e7e4bc-764b-49e0-bb92-dea358fafcc5\") " pod="openshift-ingress-canary/ingress-canary-v9wzp"
Oct 31 01:20:45.204197 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204195    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-systemd\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.204837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204241    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-run-multus-certs\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.204837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204260    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-web-config\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.204837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204275    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec3e19fc-4890-424b-86ef-f5ad1e926158-cnibin\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:20:45.204837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204317    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xs45\" (UniqueName: \"kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45\") pod \"network-check-target-jn7dx\" (UID: \"2aa13b7e-bcfc-4ca6-96bd-086caf620c9c\") " pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:20:45.204837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204358    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.204837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204378    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4cbn\" (UniqueName: \"kubernetes.io/projected/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243-kube-api-access-n4cbn\") pod \"prometheus-operator-9cd89b74c-97j2h\" (UID: \"c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h"
Oct 31 01:20:45.204837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204460    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 01:20:45.204837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204557    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-kubelet\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.204837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204580    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-cnibin\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.204837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204600    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ec3e19fc-4890-424b-86ef-f5ad1e926158-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:20:45.204837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204645    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.204837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204683    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-sysctl-d\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.204837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204752    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b7841e1-63bb-4e71-972e-1020c47d2ccb-ovnkube-config\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.204837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204785    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-etc-kubernetes\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.204837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204809    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.204837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204831    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ec3e19fc-4890-424b-86ef-f5ad1e926158-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:20:45.204837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204840    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.205503 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204911    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b7841e1-63bb-4e71-972e-1020c47d2ccb-ovnkube-config\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.205503 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204912    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzk26\" (UniqueName: \"kubernetes.io/projected/8aa9f886-74c9-442f-89fc-beb5fc0bf260-kube-api-access-gzk26\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.205503 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204956    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.205503 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.204990    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aa56c5f6-4784-4b24-9daa-d17f7b296d8b-config-volume\") pod \"dns-default-kmxk8\" (UID: \"aa56c5f6-4784-4b24-9daa-d17f7b296d8b\") " pod="openshift-dns/dns-default-kmxk8"
Oct 31 01:20:45.205503 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.205043    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:45.205503 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.205061    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.205503 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.205146    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-97j2h\" (UID: \"c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h"
Oct 31 01:20:45.205503 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.205178    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9jkq\" (UniqueName: \"kubernetes.io/projected/86947040-78ac-43d5-b932-9acd63cf43e7-kube-api-access-v9jkq\") pod \"node-resolver-xghmx\" (UID: \"86947040-78ac-43d5-b932-9acd63cf43e7\") " pod="openshift-dns/node-resolver-xghmx"
Oct 31 01:20:45.205503 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.205212    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwd6x\" (UniqueName: \"kubernetes.io/projected/db94c614-bfe5-4328-b1aa-770694e4fca1-kube-api-access-xwd6x\") pod \"iptables-alerter-lhhnx\" (UID: \"db94c614-bfe5-4328-b1aa-770694e4fca1\") " pod="openshift-network-operator/iptables-alerter-lhhnx"
Oct 31 01:20:45.205503 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.205204    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:45.205503 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.205243    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f-konnectivity-ca\") pod \"konnectivity-agent-wnmfp\" (UID: \"308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f\") " pod="kube-system/konnectivity-agent-wnmfp"
Oct 31 01:20:45.205503 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.205266    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aa56c5f6-4784-4b24-9daa-d17f7b296d8b-config-volume\") pod \"dns-default-kmxk8\" (UID: \"aa56c5f6-4784-4b24-9daa-d17f7b296d8b\") " pod="openshift-dns/dns-default-kmxk8"
Oct 31 01:20:45.205503 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.205276    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g8zb\" (UniqueName: \"kubernetes.io/projected/aa56c5f6-4784-4b24-9daa-d17f7b296d8b-kube-api-access-7g8zb\") pod \"dns-default-kmxk8\" (UID: \"aa56c5f6-4784-4b24-9daa-d17f7b296d8b\") " pod="openshift-dns/dns-default-kmxk8"
Oct 31 01:20:45.205503 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.205350    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-grpc-tls\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:45.205503 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.205377    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-node-log\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.205503 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.205382    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-97j2h\" (UID: \"c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h"
Oct 31 01:20:45.205503 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.205397    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-run-netns\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.205503 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.205436    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec3e19fc-4890-424b-86ef-f5ad1e926158-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:20:45.205503 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.205511    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pdph\" (UniqueName: \"kubernetes.io/projected/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-kube-api-access-7pdph\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:45.206390 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.205536    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.206390 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.205564    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de2a3af0-1220-4a82-a77a-772f99694285-metrics-client-ca\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:45.206390 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.205610    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-grpc-tls\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:45.206390 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.205947    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-config-out\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.206390 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.206016    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f-konnectivity-ca\") pod \"konnectivity-agent-wnmfp\" (UID: \"308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f\") " pod="kube-system/konnectivity-agent-wnmfp"
Oct 31 01:20:45.206390 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.206031    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw5bx\" (UniqueName: \"kubernetes.io/projected/52e25fa9-6c04-453a-a0b2-cca32a428a5a-kube-api-access-nw5bx\") pod \"node-ca-tzz5q\" (UID: \"52e25fa9-6c04-453a-a0b2-cca32a428a5a\") " pod="openshift-image-registry/node-ca-tzz5q"
Oct 31 01:20:45.206390 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.206063    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.206390 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.206102    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-var-lib-kubelet\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.206390 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.206132    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-hostroot\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.206390 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.206191    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-run-netns\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.206390 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.206273    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-run-openvswitch\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.206390 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.206314    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b7841e1-63bb-4e71-972e-1020c47d2ccb-env-overrides\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.206390 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.206343    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-system-cni-dir\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.206390 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.206343    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.206390 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.206191    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-config-out\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.207144 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.206606    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.207144 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.206695    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de2a3af0-1220-4a82-a77a-772f99694285-metrics-client-ca\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:45.207144 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.206694    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-accelerators-collector-config\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:45.207144 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.206774    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.207144 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.206781    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52e25fa9-6c04-453a-a0b2-cca32a428a5a-host\") pod \"node-ca-tzz5q\" (UID: \"52e25fa9-6c04-453a-a0b2-cca32a428a5a\") " pod="openshift-image-registry/node-ca-tzz5q"
Oct 31 01:20:45.207144 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.206845    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs\") pod \"network-metrics-daemon-vfk9h\" (UID: \"d9e6beef-017b-49a6-81a3-45b8faf145fc\") " pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:20:45.207144 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.206926    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-97j2h\" (UID: \"c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h"
Oct 31 01:20:45.207144 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.206963    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.207144 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.206983    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0cbe408-97f0-48da-ab7d-ab4d144f0796-tmp\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.207144 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.207023    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr95h\" (UniqueName: \"kubernetes.io/projected/ec3e19fc-4890-424b-86ef-f5ad1e926158-kube-api-access-qr95h\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:20:45.207144 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.207041    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-accelerators-collector-config\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:45.207144 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.207073    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5md27\" (UniqueName: \"kubernetes.io/projected/d9e6beef-017b-49a6-81a3-45b8faf145fc-kube-api-access-5md27\") pod \"network-metrics-daemon-vfk9h\" (UID: \"d9e6beef-017b-49a6-81a3-45b8faf145fc\") " pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:20:45.207144 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.207128    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e6beef-017b-49a6-81a3-45b8faf145fc-metrics-certs\") pod \"network-metrics-daemon-vfk9h\" (UID: \"d9e6beef-017b-49a6-81a3-45b8faf145fc\") " pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:20:45.207715 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.207156    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-97j2h\" (UID: \"c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h"
Oct 31 01:20:45.207715 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.207192    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-multus-socket-dir-parent\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.207715 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.207236    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-device-dir\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:20:45.207715 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.207275    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-crio-socket\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:45.207715 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.207281    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0cbe408-97f0-48da-ab7d-ab4d144f0796-tmp\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.207715 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.207303    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b7841e1-63bb-4e71-972e-1020c47d2ccb-env-overrides\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.207715 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.207346    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmwzw\" (UniqueName: \"kubernetes.io/projected/b0cbe408-97f0-48da-ab7d-ab4d144f0796-kube-api-access-xmwzw\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.207715 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.207395    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-systemd-units\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.207715 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.207438    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-run-ovn\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.207715 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.207483    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-log-socket\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.207715 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.207516    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.207715 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.207580    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-var-lib-cni-multus\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.207715 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.207635    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/81c10625-0368-49c7-89cf-eb51f06005d0-multus-daemon-config\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.207715 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.207694    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt84f\" (UniqueName: \"kubernetes.io/projected/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-kube-api-access-qt84f\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.208313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.207760    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b7841e1-63bb-4e71-972e-1020c47d2ccb-ovn-node-metrics-cert\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.208313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.207842    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/81c10625-0368-49c7-89cf-eb51f06005d0-multus-daemon-config\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.208313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.207900    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v74ww\" (UniqueName: \"kubernetes.io/projected/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-kube-api-access-v74ww\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:20:45.208313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.207980    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c8rn\" (UniqueName: \"kubernetes.io/projected/60e7e4bc-764b-49e0-bb92-dea358fafcc5-kube-api-access-9c8rn\") pod \"ingress-canary-v9wzp\" (UID: \"60e7e4bc-764b-49e0-bb92-dea358fafcc5\") " pod="openshift-ingress-canary/ingress-canary-v9wzp"
Oct 31 01:20:45.208313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.208017    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.208313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.208054    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-multus-conf-dir\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.208313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.208088    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.208313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.208116    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.208313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.208135    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b7841e1-63bb-4e71-972e-1020c47d2ccb-ovn-node-metrics-cert\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.208313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.208150    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa56c5f6-4784-4b24-9daa-d17f7b296d8b-tmp-dir\") pod \"dns-default-kmxk8\" (UID: \"aa56c5f6-4784-4b24-9daa-d17f7b296d8b\") " pod="openshift-dns/dns-default-kmxk8"
Oct 31 01:20:45.208313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.208204    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db94c614-bfe5-4328-b1aa-770694e4fca1-host-slash\") pod \"iptables-alerter-lhhnx\" (UID: \"db94c614-bfe5-4328-b1aa-770694e4fca1\") " pod="openshift-network-operator/iptables-alerter-lhhnx"
Oct 31 01:20:45.208313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.208238    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec3e19fc-4890-424b-86ef-f5ad1e926158-system-cni-dir\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:20:45.208853 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.208316    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.208853 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.208405    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa56c5f6-4784-4b24-9daa-d17f7b296d8b-tmp-dir\") pod \"dns-default-kmxk8\" (UID: \"aa56c5f6-4784-4b24-9daa-d17f7b296d8b\") " pod="openshift-dns/dns-default-kmxk8"
Oct 31 01:20:45.208853 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.208477    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzjfj\" (UniqueName: \"kubernetes.io/projected/de2a3af0-1220-4a82-a77a-772f99694285-kube-api-access-qzjfj\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:45.208853 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.208525    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-tls\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:45.208853 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.208557    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76b7d8bd-5205-4243-a45f-c208a65bbc30-metrics-client-ca\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:45.208853 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.208620    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.208853 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.208756    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8tn8\" (UniqueName: \"kubernetes.io/projected/81c10625-0368-49c7-89cf-eb51f06005d0-kube-api-access-v8tn8\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.208853 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.208796    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8aa9f886-74c9-442f-89fc-beb5fc0bf260-config-out\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.209191 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.208980    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.209191 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.209132    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-tls\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:45.209191 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.209142    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de2a3af0-1220-4a82-a77a-772f99694285-root\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:45.209359 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.209261    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8aa9f886-74c9-442f-89fc-beb5fc0bf260-config-out\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.209359 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.209305    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-cni-netd\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.209359 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.209345    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:45.209885 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.209742    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86947040-78ac-43d5-b932-9acd63cf43e7-tmp-dir\") pod \"node-resolver-xghmx\" (UID: \"86947040-78ac-43d5-b932-9acd63cf43e7\") " pod="openshift-dns/node-resolver-xghmx"
Oct 31 01:20:45.209885 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.209757    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76b7d8bd-5205-4243-a45f-c208a65bbc30-metrics-client-ca\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:45.209885 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.209826    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-slash\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.209885 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.209864    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-data-volume\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:45.210064 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.209922    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:45.210064 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.209934    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.210064 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.209965    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86947040-78ac-43d5-b932-9acd63cf43e7-tmp-dir\") pod \"node-resolver-xghmx\" (UID: \"86947040-78ac-43d5-b932-9acd63cf43e7\") " pod="openshift-dns/node-resolver-xghmx"
Oct 31 01:20:45.210064 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.209864    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:45.210064 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.209969    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.210064 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.210044    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-data-volume\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:45.210854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.210178    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-lib-modules\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.210854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.210253    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-etc-openvswitch\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.210854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.210298    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.210854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.210311    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b7841e1-63bb-4e71-972e-1020c47d2ccb-ovnkube-script-lib\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.210854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.210393    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-multus-cni-dir\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.210854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.210412    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:45.210854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.210418    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-wtmp\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:45.210854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.210487    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.210854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.210567    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-var-lib-cni-bin\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.210854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.210625    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:45.210854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.210659    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b7841e1-63bb-4e71-972e-1020c47d2ccb-ovnkube-script-lib\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.210854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.210667    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-sys\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.210854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.210704    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-tuned\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.210854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.210755    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-run-k8s-cni-cncf-io\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.210854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.210787    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f8449422-f48d-4df2-8548-9de1c5339fd0-dbus\") pod \"global-pull-secret-syncer-5crfw\" (UID: \"f8449422-f48d-4df2-8548-9de1c5339fd0\") " pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:20:45.211509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.210881    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.211509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.210890    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.211509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.210934    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-97j2h\" (UID: \"c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h"
Oct 31 01:20:45.211509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.210971    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-kubernetes\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.211509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.211052    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-tuned\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.211509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.211166    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.211509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.211227    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec3e19fc-4890-424b-86ef-f5ad1e926158-cni-binary-copy\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:20:45.211509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.211229    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:45.211509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.211256    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:45.211509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.211285    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-97j2h\" (UID: \"c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h"
Oct 31 01:20:45.211509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.211493    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:45.211987 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.211512    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec3e19fc-4890-424b-86ef-f5ad1e926158-cni-binary-copy\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:20:45.211987 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.211544    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.211987 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.211834    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/52e25fa9-6c04-453a-a0b2-cca32a428a5a-serviceca\") pod \"node-ca-tzz5q\" (UID: \"52e25fa9-6c04-453a-a0b2-cca32a428a5a\") " pod="openshift-image-registry/node-ca-tzz5q"
Oct 31 01:20:45.211987 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.211856    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.211987 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.211904    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-cni-bin\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.211987 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.211947    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.212246 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.212226    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8aa9f886-74c9-442f-89fc-beb5fc0bf260-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.212294 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.212269    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-textfile\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:45.212341 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.212313    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.212389 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.212346    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-run\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.212432 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.212419    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.212492 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.212473    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa56c5f6-4784-4b24-9daa-d17f7b296d8b-metrics-tls\") pod \"dns-default-kmxk8\" (UID: \"aa56c5f6-4784-4b24-9daa-d17f7b296d8b\") " pod="openshift-dns/dns-default-kmxk8"
Oct 31 01:20:45.212535 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.212506    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:45.212535 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.212509    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/52e25fa9-6c04-453a-a0b2-cca32a428a5a-serviceca\") pod \"node-ca-tzz5q\" (UID: \"52e25fa9-6c04-453a-a0b2-cca32a428a5a\") " pod="openshift-image-registry/node-ca-tzz5q"
Oct 31 01:20:45.212637 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.212537    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.212637 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.212598    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/86947040-78ac-43d5-b932-9acd63cf43e7-hosts-file\") pod \"node-resolver-xghmx\" (UID: \"86947040-78ac-43d5-b932-9acd63cf43e7\") " pod="openshift-dns/node-resolver-xghmx"
Oct 31 01:20:45.212637 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.212625    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-sysctl-conf\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.212879 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.212656    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-run-systemd\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.212879 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.212716    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret\") pod \"global-pull-secret-syncer-5crfw\" (UID: \"f8449422-f48d-4df2-8548-9de1c5339fd0\") " pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:20:45.212879 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.212748    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:20:45.212879 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.212754    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:45.212879 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.212776    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-sys-fs\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:20:45.212879 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.212805    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:45.212879 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.212835    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-modprobe-d\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.212879 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.212850    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-textfile\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:45.212879 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.212877    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-var-lib-kubelet\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.213493 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.212995    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-host\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.213493 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.213028    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-var-lib-openvswitch\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.213493 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.213176    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa56c5f6-4784-4b24-9daa-d17f7b296d8b-metrics-tls\") pod \"dns-default-kmxk8\" (UID: \"aa56c5f6-4784-4b24-9daa-d17f7b296d8b\") " pod="openshift-dns/dns-default-kmxk8"
Oct 31 01:20:45.213493 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.213244    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-run-ovn-kubernetes\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.213493 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.213233    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76b7d8bd-5205-4243-a45f-c208a65bbc30-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:45.213493 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.213297    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh6l2\" (UniqueName: \"kubernetes.io/projected/6b7841e1-63bb-4e71-972e-1020c47d2ccb-kube-api-access-kh6l2\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.213493 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.213437    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:20:45.213493 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.213463    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.213493 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.213491    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-socket-dir\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:20:45.213851 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.213520    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-registration-dir\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:20:45.213851 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.213546    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f8449422-f48d-4df2-8548-9de1c5339fd0-original-pull-secret\") pod \"global-pull-secret-syncer-5crfw\" (UID: \"f8449422-f48d-4df2-8548-9de1c5339fd0\") " pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:20:45.216465 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.216427    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 01:20:45.217190 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.217167    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xs45\" (UniqueName: \"kubernetes.io/projected/2aa13b7e-bcfc-4ca6-96bd-086caf620c9c-kube-api-access-9xs45\") pod \"network-check-target-jn7dx\" (UID: \"2aa13b7e-bcfc-4ca6-96bd-086caf620c9c\") " pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:20:45.217337 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.217318    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4cbn\" (UniqueName: \"kubernetes.io/projected/c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243-kube-api-access-n4cbn\") pod \"prometheus-operator-9cd89b74c-97j2h\" (UID: \"c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-97j2h"
Oct 31 01:20:45.217378 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.217335    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdzbp\" (UniqueName: \"kubernetes.io/projected/76b7d8bd-5205-4243-a45f-c208a65bbc30-kube-api-access-fdzbp\") pod \"thanos-querier-86d478dd89-l66f7\" (UID: \"76b7d8bd-5205-4243-a45f-c208a65bbc30\") " pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:45.217673 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.217653    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 31 01:20:45.217751 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.217736    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 31 01:20:45.218482 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.218466    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9jkq\" (UniqueName: \"kubernetes.io/projected/86947040-78ac-43d5-b932-9acd63cf43e7-kube-api-access-v9jkq\") pod \"node-resolver-xghmx\" (UID: \"86947040-78ac-43d5-b932-9acd63cf43e7\") " pod="openshift-dns/node-resolver-xghmx"
Oct 31 01:20:45.219216 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.219196    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pdph\" (UniqueName: \"kubernetes.io/projected/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-kube-api-access-7pdph\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:45.219646 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.219624    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw5bx\" (UniqueName: \"kubernetes.io/projected/52e25fa9-6c04-453a-a0b2-cca32a428a5a-kube-api-access-nw5bx\") pod \"node-ca-tzz5q\" (UID: \"52e25fa9-6c04-453a-a0b2-cca32a428a5a\") " pod="openshift-image-registry/node-ca-tzz5q"
Oct 31 01:20:45.219722 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.219682    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwd6x\" (UniqueName: \"kubernetes.io/projected/db94c614-bfe5-4328-b1aa-770694e4fca1-kube-api-access-xwd6x\") pod \"iptables-alerter-lhhnx\" (UID: \"db94c614-bfe5-4328-b1aa-770694e4fca1\") " pod="openshift-network-operator/iptables-alerter-lhhnx"
Oct 31 01:20:45.219778 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.219747    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.220349 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.220329    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzk26\" (UniqueName: \"kubernetes.io/projected/8aa9f886-74c9-442f-89fc-beb5fc0bf260-kube-api-access-gzk26\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.220604 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.220585    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g8zb\" (UniqueName: \"kubernetes.io/projected/aa56c5f6-4784-4b24-9daa-d17f7b296d8b-kube-api-access-7g8zb\") pod \"dns-default-kmxk8\" (UID: \"aa56c5f6-4784-4b24-9daa-d17f7b296d8b\") " pod="openshift-dns/dns-default-kmxk8"
Oct 31 01:20:45.224748 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.224660    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8tn8\" (UniqueName: \"kubernetes.io/projected/81c10625-0368-49c7-89cf-eb51f06005d0-kube-api-access-v8tn8\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.225073 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.225044    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa9f886-74c9-442f-89fc-beb5fc0bf260-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa9f886-74c9-442f-89fc-beb5fc0bf260\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:45.225609 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.225590    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.226403 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.226381    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c8rn\" (UniqueName: \"kubernetes.io/projected/60e7e4bc-764b-49e0-bb92-dea358fafcc5-kube-api-access-9c8rn\") pod \"ingress-canary-v9wzp\" (UID: \"60e7e4bc-764b-49e0-bb92-dea358fafcc5\") " pod="openshift-ingress-canary/ingress-canary-v9wzp"
Oct 31 01:20:45.226961 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.226943    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5md27\" (UniqueName: \"kubernetes.io/projected/d9e6beef-017b-49a6-81a3-45b8faf145fc-kube-api-access-5md27\") pod \"network-metrics-daemon-vfk9h\" (UID: \"d9e6beef-017b-49a6-81a3-45b8faf145fc\") " pod="openshift-multus/network-metrics-daemon-vfk9h"
Oct 31 01:20:45.227209 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.227192    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v74ww\" (UniqueName: \"kubernetes.io/projected/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-kube-api-access-v74ww\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:20:45.227735 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.227713    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmwzw\" (UniqueName: \"kubernetes.io/projected/b0cbe408-97f0-48da-ab7d-ab4d144f0796-kube-api-access-xmwzw\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.227973 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.227941    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr95h\" (UniqueName: \"kubernetes.io/projected/ec3e19fc-4890-424b-86ef-f5ad1e926158-kube-api-access-qr95h\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:20:45.227973 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.227948    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzjfj\" (UniqueName: \"kubernetes.io/projected/de2a3af0-1220-4a82-a77a-772f99694285-kube-api-access-qzjfj\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:45.229163 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.229138    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt84f\" (UniqueName: \"kubernetes.io/projected/0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80-kube-api-access-qt84f\") pod \"alertmanager-main-0\" (UID: \"0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:20:45.238582 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.238531    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh6l2\" (UniqueName: \"kubernetes.io/projected/6b7841e1-63bb-4e71-972e-1020c47d2ccb-kube-api-access-kh6l2\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.314087 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.313988    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-etc-kubernetes\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.314087 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314033    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-node-log\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.314087 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314053    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-run-netns\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.314087 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314075    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec3e19fc-4890-424b-86ef-f5ad1e926158-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:20:45.314400 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314107    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-etc-kubernetes\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.314400 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314127    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-node-log\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.314400 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314138    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec3e19fc-4890-424b-86ef-f5ad1e926158-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:20:45.314400 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314161    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-run-netns\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.314400 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314170    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-var-lib-kubelet\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.314400 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314189    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-hostroot\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.314400 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314205    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-run-netns\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.314400 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314230    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-run-openvswitch\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.314400 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314190    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-var-lib-kubelet\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.314400 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314253    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-system-cni-dir\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.314400 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314278    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52e25fa9-6c04-453a-a0b2-cca32a428a5a-host\") pod \"node-ca-tzz5q\" (UID: \"52e25fa9-6c04-453a-a0b2-cca32a428a5a\") " pod="openshift-image-registry/node-ca-tzz5q"
Oct 31 01:20:45.314400 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314296    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-multus-socket-dir-parent\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.314400 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314288    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-system-cni-dir\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.314400 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314334    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-device-dir\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:20:45.314400 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314358    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-crio-socket\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:45.314400 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314364    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-run-netns\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.314400 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314383    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-systemd-units\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.314400 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314394    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-run-openvswitch\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.314400 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314403    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-run-ovn\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314419    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-log-socket\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314432    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-multus-socket-dir-parent\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314436    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314469    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-var-lib-cni-multus\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314490    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-multus-conf-dir\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314493    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-hostroot\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314516    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52e25fa9-6c04-453a-a0b2-cca32a428a5a-host\") pod \"node-ca-tzz5q\" (UID: \"52e25fa9-6c04-453a-a0b2-cca32a428a5a\") " pod="openshift-image-registry/node-ca-tzz5q"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314526    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-run-ovn\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314549    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-device-dir\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314554    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ce7c59f0-b861-41ab-a5ec-4f37bba8a619-crio-socket\") pod \"insights-runtime-extractor-8n2r2\" (UID: \"ce7c59f0-b861-41ab-a5ec-4f37bba8a619\") " pod="openshift-insights/insights-runtime-extractor-8n2r2"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314560    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-systemd-units\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314581    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-var-lib-cni-multus\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314587    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db94c614-bfe5-4328-b1aa-770694e4fca1-host-slash\") pod \"iptables-alerter-lhhnx\" (UID: \"db94c614-bfe5-4328-b1aa-770694e4fca1\") " pod="openshift-network-operator/iptables-alerter-lhhnx"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314610    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-log-socket\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314614    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-multus-conf-dir\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314616    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec3e19fc-4890-424b-86ef-f5ad1e926158-system-cni-dir\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314648    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec3e19fc-4890-424b-86ef-f5ad1e926158-system-cni-dir\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314620    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db94c614-bfe5-4328-b1aa-770694e4fca1-host-slash\") pod \"iptables-alerter-lhhnx\" (UID: \"db94c614-bfe5-4328-b1aa-770694e4fca1\") " pod="openshift-network-operator/iptables-alerter-lhhnx"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314648    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de2a3af0-1220-4a82-a77a-772f99694285-root\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314686    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de2a3af0-1220-4a82-a77a-772f99694285-root\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314686    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-cni-netd\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314717    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-cni-netd\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314586    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314751    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-slash\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314717    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-slash\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314782    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-lib-modules\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314805    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-etc-openvswitch\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314823    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-multus-cni-dir\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314838    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-wtmp\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314860    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-var-lib-cni-bin\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314876    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-sys\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314891    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-etc-openvswitch\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314896    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-run-k8s-cni-cncf-io\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314914    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-run-k8s-cni-cncf-io\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314938    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f8449422-f48d-4df2-8548-9de1c5339fd0-dbus\") pod \"global-pull-secret-syncer-5crfw\" (UID: \"f8449422-f48d-4df2-8548-9de1c5339fd0\") " pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314958    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-multus-cni-dir\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314963    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-kubernetes\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314990    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de2a3af0-1220-4a82-a77a-772f99694285-node-exporter-wtmp\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.314990    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-cni-bin\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315017    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-run\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315021    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-cni-bin\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315035    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/86947040-78ac-43d5-b932-9acd63cf43e7-hosts-file\") pod \"node-resolver-xghmx\" (UID: \"86947040-78ac-43d5-b932-9acd63cf43e7\") " pod="openshift-dns/node-resolver-xghmx"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315049    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-sysctl-conf\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315079    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-run-systemd\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315105    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315119    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-sys-fs\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315135    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-modprobe-d\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315149    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-var-lib-kubelet\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315152    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-lib-modules\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315176    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-host\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315193    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-var-lib-openvswitch\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315207    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-run-ovn-kubernetes\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315228    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315243    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-socket-dir\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315243    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-kubernetes\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315273    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-registration-dir\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315283    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-socket-dir\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315308    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-var-lib-cni-bin\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315325    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de2a3af0-1220-4a82-a77a-772f99694285-sys\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315329    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-registration-dir\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315341    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f8449422-f48d-4df2-8548-9de1c5339fd0-kubelet-config\") pod \"global-pull-secret-syncer-5crfw\" (UID: \"f8449422-f48d-4df2-8548-9de1c5339fd0\") " pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315367    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f8449422-f48d-4df2-8548-9de1c5339fd0-kubelet-config\") pod \"global-pull-secret-syncer-5crfw\" (UID: \"f8449422-f48d-4df2-8548-9de1c5339fd0\") " pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315378    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-sysconfig\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315393    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-sys\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315422    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-sysconfig\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315430    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-run\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315463    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-var-lib-kubelet\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.315415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315495    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-host\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.318630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315506    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/86947040-78ac-43d5-b932-9acd63cf43e7-hosts-file\") pod \"node-resolver-xghmx\" (UID: \"86947040-78ac-43d5-b932-9acd63cf43e7\") " pod="openshift-dns/node-resolver-xghmx"
Oct 31 01:20:45.318630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315519    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-var-lib-openvswitch\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.318630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315551    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-run-ovn-kubernetes\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.318630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315205    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f8449422-f48d-4df2-8548-9de1c5339fd0-dbus\") pod \"global-pull-secret-syncer-5crfw\" (UID: \"f8449422-f48d-4df2-8548-9de1c5339fd0\") " pod="kube-system/global-pull-secret-syncer-5crfw"
Oct 31 01:20:45.318630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315585    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:20:45.318630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315607    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de2a3af0-1220-4a82-a77a-772f99694285-sys\") pod \"node-exporter-cdkr4\" (UID: \"de2a3af0-1220-4a82-a77a-772f99694285\") " pod="openshift-monitoring/node-exporter-cdkr4"
Oct 31 01:20:45.318630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315622    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-systemd\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.318630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315624    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-sysctl-conf\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.318630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315637    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-run-multus-certs\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.318630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315656    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec3e19fc-4890-424b-86ef-f5ad1e926158-cnibin\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:20:45.318630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315655    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-run-systemd\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.318630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315673    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-kubelet\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.318630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315687    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-cnibin\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.318630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315703    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-sysctl-d\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.318630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315704    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:20:45.318630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315730    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02c05e0c-b77a-45e4-b49b-1bed7e7588d0-sys-fs\") pod \"aws-ebs-csi-driver-node-x49hv\" (UID: \"02c05e0c-b77a-45e4-b49b-1bed7e7588d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x49hv"
Oct 31 01:20:45.318630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315759    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-systemd\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.318630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315780    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-host-run-multus-certs\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.318630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315790    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-modprobe-d\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.318630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315800    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec3e19fc-4890-424b-86ef-f5ad1e926158-cnibin\") pod \"multus-additional-cni-plugins-ll9gg\" (UID: \"ec3e19fc-4890-424b-86ef-f5ad1e926158\") " pod="openshift-multus/multus-additional-cni-plugins-ll9gg"
Oct 31 01:20:45.318630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315824    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81c10625-0368-49c7-89cf-eb51f06005d0-cnibin\") pod \"multus-fqpcx\" (UID: \"81c10625-0368-49c7-89cf-eb51f06005d0\") " pod="openshift-multus/multus-fqpcx"
Oct 31 01:20:45.318630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315856    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b7841e1-63bb-4e71-972e-1020c47d2ccb-host-kubelet\") pod \"ovnkube-node-tmh79\" (UID: \"6b7841e1-63bb-4e71-972e-1020c47d2ccb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:45.318630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.315869    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b0cbe408-97f0-48da-ab7d-ab4d144f0796-etc-sysctl-d\") pod \"tuned-m7xdt\" (UID: \"b0cbe408-97f0-48da-ab7d-ab4d144f0796\") " pod="openshift-cluster-node-tuning-operator/tuned-m7xdt"
Oct 31 01:20:45.430758 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.430722    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kmxk8"
Oct 31 01:20:45.603418 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.603331    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:20:45.604215 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:45.604192    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-86d478dd89-l66f7"
Oct 31 01:20:47.340520 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:47.340489    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:48.844121 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:48.844081    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:48.862874 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:48.862837    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:50.430333 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:50.430294    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:20:51.230688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:51.230649    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:20:51.230980 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:51.230951    5650 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 01:20:58.508558 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:20:58.508482    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wnmfp"
Oct 31 01:20:58.550956 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:20:58.550913    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:21:03.172822 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:21:03.172790    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tmh79"
Oct 31 01:21:16.184337 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:21:16.184297    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jn7dx"
Oct 31 01:21:28.551136 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:21:28.551096    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:21:50.445938 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:21:50.445911    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:21:58.551731 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:21:58.551686    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:22:28.551779 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:22:28.551697    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:22:58.550878 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:22:58.550834    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:23:28.551827 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:23:28.551789    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:23:58.551389 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:23:58.551308    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:24:09.628526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:09.628489    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn"]
Oct 31 01:24:09.631857 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:09.631840    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn"
Oct 31 01:24:09.638503 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:09.638422    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 01:24:09.638503 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:09.638478    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-n6k97\""
Oct 31 01:24:09.638644 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:09.638519    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 01:24:09.640851 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:09.640829    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn"]
Oct 31 01:24:09.738674 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:09.738633    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgzp9\" (UniqueName: \"kubernetes.io/projected/41976930-1143-4b6e-9a1c-83c16cfc2d5a-kube-api-access-lgzp9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn\" (UID: \"41976930-1143-4b6e-9a1c-83c16cfc2d5a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn"
Oct 31 01:24:09.738674 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:09.738676    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/41976930-1143-4b6e-9a1c-83c16cfc2d5a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn\" (UID: \"41976930-1143-4b6e-9a1c-83c16cfc2d5a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn"
Oct 31 01:24:09.738872 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:09.738792    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/41976930-1143-4b6e-9a1c-83c16cfc2d5a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn\" (UID: \"41976930-1143-4b6e-9a1c-83c16cfc2d5a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn"
Oct 31 01:24:09.839979 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:09.839942    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/41976930-1143-4b6e-9a1c-83c16cfc2d5a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn\" (UID: \"41976930-1143-4b6e-9a1c-83c16cfc2d5a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn"
Oct 31 01:24:09.840113 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:09.839999    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgzp9\" (UniqueName: \"kubernetes.io/projected/41976930-1143-4b6e-9a1c-83c16cfc2d5a-kube-api-access-lgzp9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn\" (UID: \"41976930-1143-4b6e-9a1c-83c16cfc2d5a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn"
Oct 31 01:24:09.840155 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:09.840109    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/41976930-1143-4b6e-9a1c-83c16cfc2d5a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn\" (UID: \"41976930-1143-4b6e-9a1c-83c16cfc2d5a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn"
Oct 31 01:24:09.840354 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:09.840336    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/41976930-1143-4b6e-9a1c-83c16cfc2d5a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn\" (UID: \"41976930-1143-4b6e-9a1c-83c16cfc2d5a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn"
Oct 31 01:24:09.840413 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:09.840397    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/41976930-1143-4b6e-9a1c-83c16cfc2d5a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn\" (UID: \"41976930-1143-4b6e-9a1c-83c16cfc2d5a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn"
Oct 31 01:24:09.851426 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:09.851404    5650 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 01:24:09.853613 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:09.853589    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgzp9\" (UniqueName: \"kubernetes.io/projected/41976930-1143-4b6e-9a1c-83c16cfc2d5a-kube-api-access-lgzp9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn\" (UID: \"41976930-1143-4b6e-9a1c-83c16cfc2d5a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn"
Oct 31 01:24:09.949809 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:09.949725    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn"
Oct 31 01:24:10.222790 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:10.222755    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn"]
Oct 31 01:24:10.225875 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:24:10.225839    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41976930_1143_4b6e_9a1c_83c16cfc2d5a.slice/crio-c59f2af13060e500407d300156a14ad5c28f30b2bfaab9d68675d9a9eaee636b WatchSource:0}: Error finding container c59f2af13060e500407d300156a14ad5c28f30b2bfaab9d68675d9a9eaee636b: Status 404 returned error can't find the container with id c59f2af13060e500407d300156a14ad5c28f30b2bfaab9d68675d9a9eaee636b
Oct 31 01:24:10.227984 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:10.227965    5650 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:24:10.880808 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:10.880770    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn" event={"ID":"41976930-1143-4b6e-9a1c-83c16cfc2d5a","Type":"ContainerStarted","Data":"c59f2af13060e500407d300156a14ad5c28f30b2bfaab9d68675d9a9eaee636b"}
Oct 31 01:24:11.540357 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.540324    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v"]
Oct 31 01:24:11.543858 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.543836    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v"
Oct 31 01:24:11.550546 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.550503    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-7mthc\""
Oct 31 01:24:11.554010 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.553978    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v"]
Oct 31 01:24:11.632917 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.632890    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h"]
Oct 31 01:24:11.636557 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.636529    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h"
Oct 31 01:24:11.641262 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.641232    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-z8sgf\""
Oct 31 01:24:11.645598 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.645570    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h"]
Oct 31 01:24:11.657148 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.657113    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96d392e0-a2f1-49a8-8e4d-61513f6e2558-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v\" (UID: \"96d392e0-a2f1-49a8-8e4d-61513f6e2558\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v"
Oct 31 01:24:11.657310 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.657176    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96d392e0-a2f1-49a8-8e4d-61513f6e2558-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v\" (UID: \"96d392e0-a2f1-49a8-8e4d-61513f6e2558\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v"
Oct 31 01:24:11.657310 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.657206    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njrwf\" (UniqueName: \"kubernetes.io/projected/96d392e0-a2f1-49a8-8e4d-61513f6e2558-kube-api-access-njrwf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v\" (UID: \"96d392e0-a2f1-49a8-8e4d-61513f6e2558\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v"
Oct 31 01:24:11.735548 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.735516    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67"]
Oct 31 01:24:11.739408 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.739380    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67"
Oct 31 01:24:11.744335 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.744300    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-nxzdw\""
Oct 31 01:24:11.749103 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.749073    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67"]
Oct 31 01:24:11.758671 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.758642    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0030819c-c3f8-4d08-83c9-54bb33b65180-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h\" (UID: \"0030819c-c3f8-4d08-83c9-54bb33b65180\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h"
Oct 31 01:24:11.758828 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.758693    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7prgj\" (UniqueName: \"kubernetes.io/projected/0030819c-c3f8-4d08-83c9-54bb33b65180-kube-api-access-7prgj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h\" (UID: \"0030819c-c3f8-4d08-83c9-54bb33b65180\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h"
Oct 31 01:24:11.758904 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.758825    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0030819c-c3f8-4d08-83c9-54bb33b65180-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h\" (UID: \"0030819c-c3f8-4d08-83c9-54bb33b65180\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h"
Oct 31 01:24:11.759015 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.758919    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96d392e0-a2f1-49a8-8e4d-61513f6e2558-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v\" (UID: \"96d392e0-a2f1-49a8-8e4d-61513f6e2558\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v"
Oct 31 01:24:11.759015 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.758967    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96d392e0-a2f1-49a8-8e4d-61513f6e2558-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v\" (UID: \"96d392e0-a2f1-49a8-8e4d-61513f6e2558\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v"
Oct 31 01:24:11.759015 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.758997    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njrwf\" (UniqueName: \"kubernetes.io/projected/96d392e0-a2f1-49a8-8e4d-61513f6e2558-kube-api-access-njrwf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v\" (UID: \"96d392e0-a2f1-49a8-8e4d-61513f6e2558\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v"
Oct 31 01:24:11.759362 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.759314    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96d392e0-a2f1-49a8-8e4d-61513f6e2558-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v\" (UID: \"96d392e0-a2f1-49a8-8e4d-61513f6e2558\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v"
Oct 31 01:24:11.759524 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.759383    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96d392e0-a2f1-49a8-8e4d-61513f6e2558-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v\" (UID: \"96d392e0-a2f1-49a8-8e4d-61513f6e2558\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v"
Oct 31 01:24:11.773173 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.773140    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njrwf\" (UniqueName: \"kubernetes.io/projected/96d392e0-a2f1-49a8-8e4d-61513f6e2558-kube-api-access-njrwf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v\" (UID: \"96d392e0-a2f1-49a8-8e4d-61513f6e2558\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v"
Oct 31 01:24:11.840686 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.840594    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6"]
Oct 31 01:24:11.844485 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.844430    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6"
Oct 31 01:24:11.849705 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.849673    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-9cplh\""
Oct 31 01:24:11.853722 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.853693    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6"]
Oct 31 01:24:11.859620 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.859581    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/287df43e-6f58-4044-9fd7-88d240d327d0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67\" (UID: \"287df43e-6f58-4044-9fd7-88d240d327d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67"
Oct 31 01:24:11.859786 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.859635    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0030819c-c3f8-4d08-83c9-54bb33b65180-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h\" (UID: \"0030819c-c3f8-4d08-83c9-54bb33b65180\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h"
Oct 31 01:24:11.859786 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.859716    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7prgj\" (UniqueName: \"kubernetes.io/projected/0030819c-c3f8-4d08-83c9-54bb33b65180-kube-api-access-7prgj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h\" (UID: \"0030819c-c3f8-4d08-83c9-54bb33b65180\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h"
Oct 31 01:24:11.859908 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.859785    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/287df43e-6f58-4044-9fd7-88d240d327d0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67\" (UID: \"287df43e-6f58-4044-9fd7-88d240d327d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67"
Oct 31 01:24:11.859908 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.859822    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0030819c-c3f8-4d08-83c9-54bb33b65180-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h\" (UID: \"0030819c-c3f8-4d08-83c9-54bb33b65180\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h"
Oct 31 01:24:11.859908 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.859850    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kzwg\" (UniqueName: \"kubernetes.io/projected/287df43e-6f58-4044-9fd7-88d240d327d0-kube-api-access-6kzwg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67\" (UID: \"287df43e-6f58-4044-9fd7-88d240d327d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67"
Oct 31 01:24:11.860190 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.860163    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0030819c-c3f8-4d08-83c9-54bb33b65180-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h\" (UID: \"0030819c-c3f8-4d08-83c9-54bb33b65180\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h"
Oct 31 01:24:11.860272 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.860195    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0030819c-c3f8-4d08-83c9-54bb33b65180-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h\" (UID: \"0030819c-c3f8-4d08-83c9-54bb33b65180\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h"
Oct 31 01:24:11.864954 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.864932    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v"
Oct 31 01:24:11.873746 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.873711    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7prgj\" (UniqueName: \"kubernetes.io/projected/0030819c-c3f8-4d08-83c9-54bb33b65180-kube-api-access-7prgj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h\" (UID: \"0030819c-c3f8-4d08-83c9-54bb33b65180\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h"
Oct 31 01:24:11.957821 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.957788    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h"
Oct 31 01:24:11.961188 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.961146    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2eca10cf-d21b-4f66-93c4-b8bb65733e65-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6\" (UID: \"2eca10cf-d21b-4f66-93c4-b8bb65733e65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6"
Oct 31 01:24:11.961547 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.961518    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/287df43e-6f58-4044-9fd7-88d240d327d0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67\" (UID: \"287df43e-6f58-4044-9fd7-88d240d327d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67"
Oct 31 01:24:11.961690 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.961561    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2eca10cf-d21b-4f66-93c4-b8bb65733e65-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6\" (UID: \"2eca10cf-d21b-4f66-93c4-b8bb65733e65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6"
Oct 31 01:24:11.961690 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.961647    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blkqm\" (UniqueName: \"kubernetes.io/projected/2eca10cf-d21b-4f66-93c4-b8bb65733e65-kube-api-access-blkqm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6\" (UID: \"2eca10cf-d21b-4f66-93c4-b8bb65733e65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6"
Oct 31 01:24:11.961798 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.961720    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/287df43e-6f58-4044-9fd7-88d240d327d0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67\" (UID: \"287df43e-6f58-4044-9fd7-88d240d327d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67"
Oct 31 01:24:11.961798 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.961765    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kzwg\" (UniqueName: \"kubernetes.io/projected/287df43e-6f58-4044-9fd7-88d240d327d0-kube-api-access-6kzwg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67\" (UID: \"287df43e-6f58-4044-9fd7-88d240d327d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67"
Oct 31 01:24:11.962014 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.961886    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/287df43e-6f58-4044-9fd7-88d240d327d0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67\" (UID: \"287df43e-6f58-4044-9fd7-88d240d327d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67"
Oct 31 01:24:11.962145 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.962122    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/287df43e-6f58-4044-9fd7-88d240d327d0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67\" (UID: \"287df43e-6f58-4044-9fd7-88d240d327d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67"
Oct 31 01:24:11.976100 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:11.976068    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kzwg\" (UniqueName: \"kubernetes.io/projected/287df43e-6f58-4044-9fd7-88d240d327d0-kube-api-access-6kzwg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67\" (UID: \"287df43e-6f58-4044-9fd7-88d240d327d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67"
Oct 31 01:24:12.058627 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:12.058595    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67"
Oct 31 01:24:12.062886 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:12.062758    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blkqm\" (UniqueName: \"kubernetes.io/projected/2eca10cf-d21b-4f66-93c4-b8bb65733e65-kube-api-access-blkqm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6\" (UID: \"2eca10cf-d21b-4f66-93c4-b8bb65733e65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6"
Oct 31 01:24:12.062886 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:12.062865    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2eca10cf-d21b-4f66-93c4-b8bb65733e65-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6\" (UID: \"2eca10cf-d21b-4f66-93c4-b8bb65733e65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6"
Oct 31 01:24:12.063011 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:12.062906    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2eca10cf-d21b-4f66-93c4-b8bb65733e65-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6\" (UID: \"2eca10cf-d21b-4f66-93c4-b8bb65733e65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6"
Oct 31 01:24:12.063305 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:12.063282    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2eca10cf-d21b-4f66-93c4-b8bb65733e65-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6\" (UID: \"2eca10cf-d21b-4f66-93c4-b8bb65733e65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6"
Oct 31 01:24:12.063708 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:12.063689    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2eca10cf-d21b-4f66-93c4-b8bb65733e65-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6\" (UID: \"2eca10cf-d21b-4f66-93c4-b8bb65733e65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6"
Oct 31 01:24:12.077279 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:12.077247    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blkqm\" (UniqueName: \"kubernetes.io/projected/2eca10cf-d21b-4f66-93c4-b8bb65733e65-kube-api-access-blkqm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6\" (UID: \"2eca10cf-d21b-4f66-93c4-b8bb65733e65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6"
Oct 31 01:24:12.152875 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:12.152835    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v"]
Oct 31 01:24:12.165820 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:12.165798    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6"
Oct 31 01:24:12.242696 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:12.242664    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h"]
Oct 31 01:24:12.246427 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:24:12.246387    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0030819c_c3f8_4d08_83c9_54bb33b65180.slice/crio-0105822c298310bfb7d1ca12dddd8e1226202ccca580ab05e21ee201e4c1b1f7 WatchSource:0}: Error finding container 0105822c298310bfb7d1ca12dddd8e1226202ccca580ab05e21ee201e4c1b1f7: Status 404 returned error can't find the container with id 0105822c298310bfb7d1ca12dddd8e1226202ccca580ab05e21ee201e4c1b1f7
Oct 31 01:24:12.343034 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:12.342998    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67"]
Oct 31 01:24:12.346303 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:24:12.346272    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod287df43e_6f58_4044_9fd7_88d240d327d0.slice/crio-33e262a38dabbd2e1ee36a96ec9188d34d716cc319daf4a7b9a4abcb284932fc WatchSource:0}: Error finding container 33e262a38dabbd2e1ee36a96ec9188d34d716cc319daf4a7b9a4abcb284932fc: Status 404 returned error can't find the container with id 33e262a38dabbd2e1ee36a96ec9188d34d716cc319daf4a7b9a4abcb284932fc
Oct 31 01:24:12.445215 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:12.445185    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6"]
Oct 31 01:24:12.448038 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:24:12.448006    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eca10cf_d21b_4f66_93c4_b8bb65733e65.slice/crio-6f721da8f65871f5583d62ca64df0397f6d3fc891f16e2dc0d04fa54553d7829 WatchSource:0}: Error finding container 6f721da8f65871f5583d62ca64df0397f6d3fc891f16e2dc0d04fa54553d7829: Status 404 returned error can't find the container with id 6f721da8f65871f5583d62ca64df0397f6d3fc891f16e2dc0d04fa54553d7829
Oct 31 01:24:12.894721 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:12.894651    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h" event={"ID":"0030819c-c3f8-4d08-83c9-54bb33b65180","Type":"ContainerStarted","Data":"0105822c298310bfb7d1ca12dddd8e1226202ccca580ab05e21ee201e4c1b1f7"}
Oct 31 01:24:12.900643 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:12.900560    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67" event={"ID":"287df43e-6f58-4044-9fd7-88d240d327d0","Type":"ContainerStarted","Data":"33e262a38dabbd2e1ee36a96ec9188d34d716cc319daf4a7b9a4abcb284932fc"}
Oct 31 01:24:12.905785 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:12.905705    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6" event={"ID":"2eca10cf-d21b-4f66-93c4-b8bb65733e65","Type":"ContainerStarted","Data":"6f721da8f65871f5583d62ca64df0397f6d3fc891f16e2dc0d04fa54553d7829"}
Oct 31 01:24:12.909025 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:12.908940    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v" event={"ID":"96d392e0-a2f1-49a8-8e4d-61513f6e2558","Type":"ContainerStarted","Data":"8c9903b56e26da4c592fc480ba9e90313b0a122772000b9f5ad7fe5369b7295b"}
Oct 31 01:24:13.249669 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.247690    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt"]
Oct 31 01:24:13.252628 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.251909    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt"
Oct 31 01:24:13.266153 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.266121    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-skt79\""
Oct 31 01:24:13.267130 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.267101    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt"]
Oct 31 01:24:13.374842 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.374797    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b1bcf70-2690-47c4-9ef4-c884b38c8bd8-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt\" (UID: \"6b1bcf70-2690-47c4-9ef4-c884b38c8bd8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt"
Oct 31 01:24:13.375035 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.374876    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b1bcf70-2690-47c4-9ef4-c884b38c8bd8-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt\" (UID: \"6b1bcf70-2690-47c4-9ef4-c884b38c8bd8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt"
Oct 31 01:24:13.375035 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.374909    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfrdz\" (UniqueName: \"kubernetes.io/projected/6b1bcf70-2690-47c4-9ef4-c884b38c8bd8-kube-api-access-sfrdz\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt\" (UID: \"6b1bcf70-2690-47c4-9ef4-c884b38c8bd8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt"
Oct 31 01:24:13.477374 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.476214    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b1bcf70-2690-47c4-9ef4-c884b38c8bd8-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt\" (UID: \"6b1bcf70-2690-47c4-9ef4-c884b38c8bd8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt"
Oct 31 01:24:13.477374 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.476291    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b1bcf70-2690-47c4-9ef4-c884b38c8bd8-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt\" (UID: \"6b1bcf70-2690-47c4-9ef4-c884b38c8bd8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt"
Oct 31 01:24:13.477374 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.476322    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfrdz\" (UniqueName: \"kubernetes.io/projected/6b1bcf70-2690-47c4-9ef4-c884b38c8bd8-kube-api-access-sfrdz\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt\" (UID: \"6b1bcf70-2690-47c4-9ef4-c884b38c8bd8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt"
Oct 31 01:24:13.477374 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.477081    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b1bcf70-2690-47c4-9ef4-c884b38c8bd8-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt\" (UID: \"6b1bcf70-2690-47c4-9ef4-c884b38c8bd8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt"
Oct 31 01:24:13.477374 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.477320    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b1bcf70-2690-47c4-9ef4-c884b38c8bd8-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt\" (UID: \"6b1bcf70-2690-47c4-9ef4-c884b38c8bd8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt"
Oct 31 01:24:13.497647 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.497542    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfrdz\" (UniqueName: \"kubernetes.io/projected/6b1bcf70-2690-47c4-9ef4-c884b38c8bd8-kube-api-access-sfrdz\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt\" (UID: \"6b1bcf70-2690-47c4-9ef4-c884b38c8bd8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt"
Oct 31 01:24:13.574927 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.574851    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt"
Oct 31 01:24:13.758343 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.758295    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"]
Oct 31 01:24:13.772821 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.772786    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.787693 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.787471    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"]
Oct 31 01:24:13.790805 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.790775    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-zzksk\""
Oct 31 01:24:13.792674 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.792647    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 01:24:13.792809 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.792774    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 01:24:13.792873 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.792839    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 01:24:13.879990 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.879906    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.879990 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.879959    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.880177 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.880000    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.880177 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.880058    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.880177 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.880098    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stcrd\" (UniqueName: \"kubernetes.io/projected/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-kube-api-access-stcrd\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.880177 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.880136    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.880356 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.880224    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.880356 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.880255    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.880356 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.880317    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.890066 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.889973    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt"]
Oct 31 01:24:13.895787 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:24:13.895731    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b1bcf70_2690_47c4_9ef4_c884b38c8bd8.slice/crio-361a3c3aa64be2342f1fa4d3c0bc5659a4a7a49d062f16fb88e0b565bf369dbd WatchSource:0}: Error finding container 361a3c3aa64be2342f1fa4d3c0bc5659a4a7a49d062f16fb88e0b565bf369dbd: Status 404 returned error can't find the container with id 361a3c3aa64be2342f1fa4d3c0bc5659a4a7a49d062f16fb88e0b565bf369dbd
Oct 31 01:24:13.916849 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.916752    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt" event={"ID":"6b1bcf70-2690-47c4-9ef4-c884b38c8bd8","Type":"ContainerStarted","Data":"361a3c3aa64be2342f1fa4d3c0bc5659a4a7a49d062f16fb88e0b565bf369dbd"}
Oct 31 01:24:13.981729 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.981161    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.981729 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.981302    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.981729 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.981346    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.981729 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.981378    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.981729 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.981418    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stcrd\" (UniqueName: \"kubernetes.io/projected/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-kube-api-access-stcrd\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.981729 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.981472    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.981729 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.981581    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.981729 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.981613    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.981729 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.981638    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.981729 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.981719    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.982385 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.982066    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.982385 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.982150    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.982385 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.982327    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.982573 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.982504    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.984599 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.984504    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.986246 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.986184    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:13.999752 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:13.999675    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:14.007008 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:14.006935    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stcrd\" (UniqueName: \"kubernetes.io/projected/64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f-kube-api-access-stcrd\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q\" (UID: \"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:14.097244 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:14.097150    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:14.425987 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:14.425846    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"]
Oct 31 01:24:14.442005 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:24:14.441965    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c1826b_d7f2_4ef7_8d5b_4c1b4a7d5c3f.slice/crio-de059a9c846e0258b58375ee602d21cc1eb8cc37f6dcede2d92a6974a6b6f70a WatchSource:0}: Error finding container de059a9c846e0258b58375ee602d21cc1eb8cc37f6dcede2d92a6974a6b6f70a: Status 404 returned error can't find the container with id de059a9c846e0258b58375ee602d21cc1eb8cc37f6dcede2d92a6974a6b6f70a
Oct 31 01:24:14.925881 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:14.925835    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q" event={"ID":"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f","Type":"ContainerStarted","Data":"de059a9c846e0258b58375ee602d21cc1eb8cc37f6dcede2d92a6974a6b6f70a"}
Oct 31 01:24:21.201338 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:21.201303    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:24:21.201791 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:21.201397    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:24:21.201791 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:21.201456    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:24:21.982112 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:21.981682    5650 generic.go:356] "Generic (PLEG): container finished" podID="41976930-1143-4b6e-9a1c-83c16cfc2d5a" containerID="26820de7301f0c7bb27f973b21cb2352f5c661c26db1fc97a8898f3c12eb91b3" exitCode=0
Oct 31 01:24:21.982112 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:21.981801    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn" event={"ID":"41976930-1143-4b6e-9a1c-83c16cfc2d5a","Type":"ContainerDied","Data":"26820de7301f0c7bb27f973b21cb2352f5c661c26db1fc97a8898f3c12eb91b3"}
Oct 31 01:24:21.987570 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:21.986939    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q" event={"ID":"64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f","Type":"ContainerStarted","Data":"698d57558893bb91b2621f57e3505c77af538429cf89eb02aa326e9a3c676028"}
Oct 31 01:24:22.098357 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:22.098264    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:22.104240 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:22.104208    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:22.135583 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:22.135517    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q" podStartSLOduration=2.385626635 podStartE2EDuration="9.135500393s" totalImagesPullingTime="6.749873758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:22.033419948 +0000 UTC m=+218.037134238" watchObservedRunningTime="2026-10-31 01:24:22.135500393 +0000 UTC m=+218.139214678"
Oct 31 01:24:22.995739 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:22.995551    5650 generic.go:356] "Generic (PLEG): container finished" podID="41976930-1143-4b6e-9a1c-83c16cfc2d5a" containerID="6b960e79e876877ea81619cbfe39f74fb9550a8277db6a78aa8041165913ecf8" exitCode=0
Oct 31 01:24:22.995739 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:22.995635    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn" event={"ID":"41976930-1143-4b6e-9a1c-83c16cfc2d5a","Type":"ContainerDied","Data":"6b960e79e876877ea81619cbfe39f74fb9550a8277db6a78aa8041165913ecf8"}
Oct 31 01:24:23.000083 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:22.998927    5650 generic.go:356] "Generic (PLEG): container finished" podID="6b1bcf70-2690-47c4-9ef4-c884b38c8bd8" containerID="bea1ad922039927ed89d52d1f0a54e02ac023e5d80c916eadf44b44ab06b997c" exitCode=0
Oct 31 01:24:23.000083 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:22.999581    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt" event={"ID":"6b1bcf70-2690-47c4-9ef4-c884b38c8bd8","Type":"ContainerDied","Data":"bea1ad922039927ed89d52d1f0a54e02ac023e5d80c916eadf44b44ab06b997c"}
Oct 31 01:24:23.000083 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:22.999634    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:23.001168 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:23.001018    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q"
Oct 31 01:24:25.012968 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:25.012931    5650 generic.go:356] "Generic (PLEG): container finished" podID="0030819c-c3f8-4d08-83c9-54bb33b65180" containerID="446d0b76420d91163be9c75ee0c17662cdffd4a7f960b74850be769a4a0ff8a6" exitCode=0
Oct 31 01:24:25.013507 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:25.013042    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h" event={"ID":"0030819c-c3f8-4d08-83c9-54bb33b65180","Type":"ContainerDied","Data":"446d0b76420d91163be9c75ee0c17662cdffd4a7f960b74850be769a4a0ff8a6"}
Oct 31 01:24:25.020019 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:25.018177    5650 generic.go:356] "Generic (PLEG): container finished" podID="96d392e0-a2f1-49a8-8e4d-61513f6e2558" containerID="9fb03e2804a321b2ca0616dfbc7b6d475966a6329fdb9ba24ba6b5797ed80f17" exitCode=0
Oct 31 01:24:25.020019 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:25.018302    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v" event={"ID":"96d392e0-a2f1-49a8-8e4d-61513f6e2558","Type":"ContainerDied","Data":"9fb03e2804a321b2ca0616dfbc7b6d475966a6329fdb9ba24ba6b5797ed80f17"}
Oct 31 01:24:25.022570 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:25.022544    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67" event={"ID":"287df43e-6f58-4044-9fd7-88d240d327d0","Type":"ContainerStarted","Data":"cbd01c7e67cd4b05444651a8581d020c7f3f9132cc87c24c9521bc92a9639ca6"}
Oct 31 01:24:26.029999 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:26.029956    5650 generic.go:356] "Generic (PLEG): container finished" podID="0030819c-c3f8-4d08-83c9-54bb33b65180" containerID="f712250104d86591aa5f8ac8b21344a8491620d93f33c583787f5f24e65198cf" exitCode=0
Oct 31 01:24:26.030537 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:26.030066    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h" event={"ID":"0030819c-c3f8-4d08-83c9-54bb33b65180","Type":"ContainerDied","Data":"f712250104d86591aa5f8ac8b21344a8491620d93f33c583787f5f24e65198cf"}
Oct 31 01:24:26.032566 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:26.032504    5650 generic.go:356] "Generic (PLEG): container finished" podID="2eca10cf-d21b-4f66-93c4-b8bb65733e65" containerID="21b4d34e1fc65ccc7c006d88b667dec4910839591807ccaf2b1fcbcf5ae53ef8" exitCode=0
Oct 31 01:24:26.033391 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:26.032685    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6" event={"ID":"2eca10cf-d21b-4f66-93c4-b8bb65733e65","Type":"ContainerDied","Data":"21b4d34e1fc65ccc7c006d88b667dec4910839591807ccaf2b1fcbcf5ae53ef8"}
Oct 31 01:24:26.036658 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:26.036490    5650 generic.go:356] "Generic (PLEG): container finished" podID="287df43e-6f58-4044-9fd7-88d240d327d0" containerID="cbd01c7e67cd4b05444651a8581d020c7f3f9132cc87c24c9521bc92a9639ca6" exitCode=0
Oct 31 01:24:26.036658 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:26.036527    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67" event={"ID":"287df43e-6f58-4044-9fd7-88d240d327d0","Type":"ContainerDied","Data":"cbd01c7e67cd4b05444651a8581d020c7f3f9132cc87c24c9521bc92a9639ca6"}
Oct 31 01:24:27.044582 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:27.044530    5650 generic.go:356] "Generic (PLEG): container finished" podID="6b1bcf70-2690-47c4-9ef4-c884b38c8bd8" containerID="bb4d66956a9024c120f69414da8bcd7c713b377e0581e55623cefaa9186d8450" exitCode=0
Oct 31 01:24:27.045025 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:27.044657    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt" event={"ID":"6b1bcf70-2690-47c4-9ef4-c884b38c8bd8","Type":"ContainerDied","Data":"bb4d66956a9024c120f69414da8bcd7c713b377e0581e55623cefaa9186d8450"}
Oct 31 01:24:27.049904 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:27.049108    5650 generic.go:356] "Generic (PLEG): container finished" podID="96d392e0-a2f1-49a8-8e4d-61513f6e2558" containerID="6c0b402a012838902359ceef5e6d0ae3f1ed38da112050571485ac39f7ffbe11" exitCode=0
Oct 31 01:24:27.049904 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:27.049177    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v" event={"ID":"96d392e0-a2f1-49a8-8e4d-61513f6e2558","Type":"ContainerDied","Data":"6c0b402a012838902359ceef5e6d0ae3f1ed38da112050571485ac39f7ffbe11"}
Oct 31 01:24:28.055882 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:28.055840    5650 generic.go:356] "Generic (PLEG): container finished" podID="287df43e-6f58-4044-9fd7-88d240d327d0" containerID="9870a6b5d8b2442e701ad3274ba6ffd9e73750989797759671bdbe65719ad4ad" exitCode=0
Oct 31 01:24:28.056364 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:28.056186    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67" event={"ID":"287df43e-6f58-4044-9fd7-88d240d327d0","Type":"ContainerDied","Data":"9870a6b5d8b2442e701ad3274ba6ffd9e73750989797759671bdbe65719ad4ad"}
Oct 31 01:24:28.059824 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:28.059773    5650 generic.go:356] "Generic (PLEG): container finished" podID="2eca10cf-d21b-4f66-93c4-b8bb65733e65" containerID="ccc50c5c894512609ccd1b22dc0e68a03d2506ac872b6ff6eaf7f4d412311a34" exitCode=0
Oct 31 01:24:28.059925 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:28.059819    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6" event={"ID":"2eca10cf-d21b-4f66-93c4-b8bb65733e65","Type":"ContainerDied","Data":"ccc50c5c894512609ccd1b22dc0e68a03d2506ac872b6ff6eaf7f4d412311a34"}
Oct 31 01:24:28.567119 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:24:28.567044    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:24:34.101021 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:34.100985    5650 generic.go:356] "Generic (PLEG): container finished" podID="41976930-1143-4b6e-9a1c-83c16cfc2d5a" containerID="639bf6a2d932cf851911c51dd9710a760a5538277fb3f687a58cba9376a00aea" exitCode=0
Oct 31 01:24:34.101549 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:34.101173    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn" event={"ID":"41976930-1143-4b6e-9a1c-83c16cfc2d5a","Type":"ContainerDied","Data":"639bf6a2d932cf851911c51dd9710a760a5538277fb3f687a58cba9376a00aea"}
Oct 31 01:24:34.104140 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:34.104112    5650 generic.go:356] "Generic (PLEG): container finished" podID="6b1bcf70-2690-47c4-9ef4-c884b38c8bd8" containerID="db6da3f610445056b2d640591af1c7470be2f7cc0ecdab58adaafb836e7a20b9" exitCode=0
Oct 31 01:24:34.104268 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:34.104204    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt" event={"ID":"6b1bcf70-2690-47c4-9ef4-c884b38c8bd8","Type":"ContainerDied","Data":"db6da3f610445056b2d640591af1c7470be2f7cc0ecdab58adaafb836e7a20b9"}
Oct 31 01:24:34.108224 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:34.107979    5650 generic.go:356] "Generic (PLEG): container finished" podID="0030819c-c3f8-4d08-83c9-54bb33b65180" containerID="2960eca3d2de1481dbe24d62f85cdd99b8025c21b2a73269e39934b8cdbf2c3d" exitCode=0
Oct 31 01:24:34.108224 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:34.108084    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h" event={"ID":"0030819c-c3f8-4d08-83c9-54bb33b65180","Type":"ContainerDied","Data":"2960eca3d2de1481dbe24d62f85cdd99b8025c21b2a73269e39934b8cdbf2c3d"}
Oct 31 01:24:35.117864 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.117831    5650 generic.go:356] "Generic (PLEG): container finished" podID="287df43e-6f58-4044-9fd7-88d240d327d0" containerID="5f65c4653be0adf36b92105b1c7fb0cf981c0178bdc92035cd5787e1b2218c7c" exitCode=0
Oct 31 01:24:35.118343 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.118270    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67" event={"ID":"287df43e-6f58-4044-9fd7-88d240d327d0","Type":"ContainerDied","Data":"5f65c4653be0adf36b92105b1c7fb0cf981c0178bdc92035cd5787e1b2218c7c"}
Oct 31 01:24:35.306124 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.306049    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn"
Oct 31 01:24:35.309750 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.309722    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h"
Oct 31 01:24:35.340831 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.340808    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt"
Oct 31 01:24:35.411074 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.411037    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0030819c-c3f8-4d08-83c9-54bb33b65180-util\" (UniqueName: \"kubernetes.io/empty-dir/0030819c-c3f8-4d08-83c9-54bb33b65180-util\") pod \"0030819c-c3f8-4d08-83c9-54bb33b65180\" (UID: \"0030819c-c3f8-4d08-83c9-54bb33b65180\") "
Oct 31 01:24:35.411261 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.411085    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41976930-1143-4b6e-9a1c-83c16cfc2d5a-util\" (UniqueName: \"kubernetes.io/empty-dir/41976930-1143-4b6e-9a1c-83c16cfc2d5a-util\") pod \"41976930-1143-4b6e-9a1c-83c16cfc2d5a\" (UID: \"41976930-1143-4b6e-9a1c-83c16cfc2d5a\") "
Oct 31 01:24:35.411261 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.411109    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41976930-1143-4b6e-9a1c-83c16cfc2d5a-kube-api-access-lgzp9\" (UniqueName: \"kubernetes.io/projected/41976930-1143-4b6e-9a1c-83c16cfc2d5a-kube-api-access-lgzp9\") pod \"41976930-1143-4b6e-9a1c-83c16cfc2d5a\" (UID: \"41976930-1143-4b6e-9a1c-83c16cfc2d5a\") "
Oct 31 01:24:35.411261 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.411169    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41976930-1143-4b6e-9a1c-83c16cfc2d5a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/41976930-1143-4b6e-9a1c-83c16cfc2d5a-bundle\") pod \"41976930-1143-4b6e-9a1c-83c16cfc2d5a\" (UID: \"41976930-1143-4b6e-9a1c-83c16cfc2d5a\") "
Oct 31 01:24:35.411261 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.411189    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0030819c-c3f8-4d08-83c9-54bb33b65180-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0030819c-c3f8-4d08-83c9-54bb33b65180-bundle\") pod \"0030819c-c3f8-4d08-83c9-54bb33b65180\" (UID: \"0030819c-c3f8-4d08-83c9-54bb33b65180\") "
Oct 31 01:24:35.411261 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.411248    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0030819c-c3f8-4d08-83c9-54bb33b65180-kube-api-access-7prgj\" (UniqueName: \"kubernetes.io/projected/0030819c-c3f8-4d08-83c9-54bb33b65180-kube-api-access-7prgj\") pod \"0030819c-c3f8-4d08-83c9-54bb33b65180\" (UID: \"0030819c-c3f8-4d08-83c9-54bb33b65180\") "
Oct 31 01:24:35.411773 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.411738    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0030819c-c3f8-4d08-83c9-54bb33b65180-bundle" pod "0030819c-c3f8-4d08-83c9-54bb33b65180" (UID: "0030819c-c3f8-4d08-83c9-54bb33b65180"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:35.411886 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.411850    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41976930-1143-4b6e-9a1c-83c16cfc2d5a-bundle" pod "41976930-1143-4b6e-9a1c-83c16cfc2d5a" (UID: "41976930-1143-4b6e-9a1c-83c16cfc2d5a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:35.413493 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.413436    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41976930-1143-4b6e-9a1c-83c16cfc2d5a-kube-api-access-lgzp9" pod "41976930-1143-4b6e-9a1c-83c16cfc2d5a" (UID: "41976930-1143-4b6e-9a1c-83c16cfc2d5a"). InnerVolumeSpecName "kube-api-access-lgzp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:35.414016 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.413994    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0030819c-c3f8-4d08-83c9-54bb33b65180-kube-api-access-7prgj" pod "0030819c-c3f8-4d08-83c9-54bb33b65180" (UID: "0030819c-c3f8-4d08-83c9-54bb33b65180"). InnerVolumeSpecName "kube-api-access-7prgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:35.417726 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.417696    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41976930-1143-4b6e-9a1c-83c16cfc2d5a-util" pod "41976930-1143-4b6e-9a1c-83c16cfc2d5a" (UID: "41976930-1143-4b6e-9a1c-83c16cfc2d5a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:35.418880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.418662    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0030819c-c3f8-4d08-83c9-54bb33b65180-util" pod "0030819c-c3f8-4d08-83c9-54bb33b65180" (UID: "0030819c-c3f8-4d08-83c9-54bb33b65180"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:35.512196 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.512158    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b1bcf70-2690-47c4-9ef4-c884b38c8bd8-util\" (UniqueName: \"kubernetes.io/empty-dir/6b1bcf70-2690-47c4-9ef4-c884b38c8bd8-util\") pod \"6b1bcf70-2690-47c4-9ef4-c884b38c8bd8\" (UID: \"6b1bcf70-2690-47c4-9ef4-c884b38c8bd8\") "
Oct 31 01:24:35.512386 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.512208    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b1bcf70-2690-47c4-9ef4-c884b38c8bd8-kube-api-access-sfrdz\" (UniqueName: \"kubernetes.io/projected/6b1bcf70-2690-47c4-9ef4-c884b38c8bd8-kube-api-access-sfrdz\") pod \"6b1bcf70-2690-47c4-9ef4-c884b38c8bd8\" (UID: \"6b1bcf70-2690-47c4-9ef4-c884b38c8bd8\") "
Oct 31 01:24:35.512386 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.512245    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b1bcf70-2690-47c4-9ef4-c884b38c8bd8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b1bcf70-2690-47c4-9ef4-c884b38c8bd8-bundle\") pod \"6b1bcf70-2690-47c4-9ef4-c884b38c8bd8\" (UID: \"6b1bcf70-2690-47c4-9ef4-c884b38c8bd8\") "
Oct 31 01:24:35.512386 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.512380    5650 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0030819c-c3f8-4d08-83c9-54bb33b65180-util\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:35.512568 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.512391    5650 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/41976930-1143-4b6e-9a1c-83c16cfc2d5a-util\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:35.512568 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.512401    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgzp9\" (UniqueName: \"kubernetes.io/projected/41976930-1143-4b6e-9a1c-83c16cfc2d5a-kube-api-access-lgzp9\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:35.512568 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.512409    5650 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/41976930-1143-4b6e-9a1c-83c16cfc2d5a-bundle\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:35.512568 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.512418    5650 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0030819c-c3f8-4d08-83c9-54bb33b65180-bundle\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:35.512568 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.512427    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7prgj\" (UniqueName: \"kubernetes.io/projected/0030819c-c3f8-4d08-83c9-54bb33b65180-kube-api-access-7prgj\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:35.512769 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.512667    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b1bcf70-2690-47c4-9ef4-c884b38c8bd8-bundle" pod "6b1bcf70-2690-47c4-9ef4-c884b38c8bd8" (UID: "6b1bcf70-2690-47c4-9ef4-c884b38c8bd8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:35.514396 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.514372    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b1bcf70-2690-47c4-9ef4-c884b38c8bd8-kube-api-access-sfrdz" pod "6b1bcf70-2690-47c4-9ef4-c884b38c8bd8" (UID: "6b1bcf70-2690-47c4-9ef4-c884b38c8bd8"). InnerVolumeSpecName "kube-api-access-sfrdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:35.517167 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.517146    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b1bcf70-2690-47c4-9ef4-c884b38c8bd8-util" pod "6b1bcf70-2690-47c4-9ef4-c884b38c8bd8" (UID: "6b1bcf70-2690-47c4-9ef4-c884b38c8bd8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:35.613804 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.613707    5650 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b1bcf70-2690-47c4-9ef4-c884b38c8bd8-util\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:35.613804 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.613747    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfrdz\" (UniqueName: \"kubernetes.io/projected/6b1bcf70-2690-47c4-9ef4-c884b38c8bd8-kube-api-access-sfrdz\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:35.613804 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:35.613762    5650 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b1bcf70-2690-47c4-9ef4-c884b38c8bd8-bundle\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:36.132551 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:36.132524    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn"
Oct 31 01:24:36.132551 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:36.132535    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6pdn" event={"ID":"41976930-1143-4b6e-9a1c-83c16cfc2d5a","Type":"ContainerDied","Data":"c59f2af13060e500407d300156a14ad5c28f30b2bfaab9d68675d9a9eaee636b"}
Oct 31 01:24:36.132979 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:36.132580    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c59f2af13060e500407d300156a14ad5c28f30b2bfaab9d68675d9a9eaee636b"
Oct 31 01:24:36.134806 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:36.134779    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt" event={"ID":"6b1bcf70-2690-47c4-9ef4-c884b38c8bd8","Type":"ContainerDied","Data":"361a3c3aa64be2342f1fa4d3c0bc5659a4a7a49d062f16fb88e0b565bf369dbd"}
Oct 31 01:24:36.134901 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:36.134825    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="361a3c3aa64be2342f1fa4d3c0bc5659a4a7a49d062f16fb88e0b565bf369dbd"
Oct 31 01:24:36.134901 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:36.134890    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zz4zt"
Oct 31 01:24:36.138228 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:36.138210    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h"
Oct 31 01:24:36.138412 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:36.138219    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6n2h" event={"ID":"0030819c-c3f8-4d08-83c9-54bb33b65180","Type":"ContainerDied","Data":"0105822c298310bfb7d1ca12dddd8e1226202ccca580ab05e21ee201e4c1b1f7"}
Oct 31 01:24:36.138553 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:36.138429    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0105822c298310bfb7d1ca12dddd8e1226202ccca580ab05e21ee201e4c1b1f7"
Oct 31 01:24:36.140921 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:36.140894    5650 generic.go:356] "Generic (PLEG): container finished" podID="96d392e0-a2f1-49a8-8e4d-61513f6e2558" containerID="d47d76971c2cef799dc0b8fdeeb4ad6cefcf542f90596b5e7bd2682b425e4ba0" exitCode=0
Oct 31 01:24:36.141029 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:36.140982    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v" event={"ID":"96d392e0-a2f1-49a8-8e4d-61513f6e2558","Type":"ContainerDied","Data":"d47d76971c2cef799dc0b8fdeeb4ad6cefcf542f90596b5e7bd2682b425e4ba0"}
Oct 31 01:24:36.283406 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:36.283378    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67"
Oct 31 01:24:36.422035 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:36.421950    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/287df43e-6f58-4044-9fd7-88d240d327d0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/287df43e-6f58-4044-9fd7-88d240d327d0-bundle\") pod \"287df43e-6f58-4044-9fd7-88d240d327d0\" (UID: \"287df43e-6f58-4044-9fd7-88d240d327d0\") "
Oct 31 01:24:36.422201 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:36.422086    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/287df43e-6f58-4044-9fd7-88d240d327d0-util\" (UniqueName: \"kubernetes.io/empty-dir/287df43e-6f58-4044-9fd7-88d240d327d0-util\") pod \"287df43e-6f58-4044-9fd7-88d240d327d0\" (UID: \"287df43e-6f58-4044-9fd7-88d240d327d0\") "
Oct 31 01:24:36.422201 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:36.422118    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/287df43e-6f58-4044-9fd7-88d240d327d0-kube-api-access-6kzwg\" (UniqueName: \"kubernetes.io/projected/287df43e-6f58-4044-9fd7-88d240d327d0-kube-api-access-6kzwg\") pod \"287df43e-6f58-4044-9fd7-88d240d327d0\" (UID: \"287df43e-6f58-4044-9fd7-88d240d327d0\") "
Oct 31 01:24:36.422787 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:36.422750    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/287df43e-6f58-4044-9fd7-88d240d327d0-bundle" pod "287df43e-6f58-4044-9fd7-88d240d327d0" (UID: "287df43e-6f58-4044-9fd7-88d240d327d0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:36.424420 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:36.424386    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/287df43e-6f58-4044-9fd7-88d240d327d0-kube-api-access-6kzwg" pod "287df43e-6f58-4044-9fd7-88d240d327d0" (UID: "287df43e-6f58-4044-9fd7-88d240d327d0"). InnerVolumeSpecName "kube-api-access-6kzwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:36.427513 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:36.427471    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/287df43e-6f58-4044-9fd7-88d240d327d0-util" pod "287df43e-6f58-4044-9fd7-88d240d327d0" (UID: "287df43e-6f58-4044-9fd7-88d240d327d0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:36.522989 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:36.522943    5650 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/287df43e-6f58-4044-9fd7-88d240d327d0-util\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:36.522989 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:36.522980    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kzwg\" (UniqueName: \"kubernetes.io/projected/287df43e-6f58-4044-9fd7-88d240d327d0-kube-api-access-6kzwg\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:36.522989 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:36.522992    5650 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/287df43e-6f58-4044-9fd7-88d240d327d0-bundle\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:37.107184 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107105    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2sb2w"]
Oct 31 01:24:37.107439 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107426    5650 cpu_manager.go:414] "containerMap: removing container" podUID="41976930-1143-4b6e-9a1c-83c16cfc2d5a" containerName="util"
Oct 31 01:24:37.107439 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107440    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41976930-1143-4b6e-9a1c-83c16cfc2d5a" containerName="util"
Oct 31 01:24:37.107556 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107472    5650 cpu_manager.go:414] "containerMap: removing container" podUID="287df43e-6f58-4044-9fd7-88d240d327d0" containerName="util"
Oct 31 01:24:37.107556 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107477    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="287df43e-6f58-4044-9fd7-88d240d327d0" containerName="util"
Oct 31 01:24:37.107556 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107489    5650 cpu_manager.go:414] "containerMap: removing container" podUID="41976930-1143-4b6e-9a1c-83c16cfc2d5a" containerName="pull"
Oct 31 01:24:37.107556 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107494    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41976930-1143-4b6e-9a1c-83c16cfc2d5a" containerName="pull"
Oct 31 01:24:37.107556 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107503    5650 cpu_manager.go:414] "containerMap: removing container" podUID="0030819c-c3f8-4d08-83c9-54bb33b65180" containerName="pull"
Oct 31 01:24:37.107556 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107507    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0030819c-c3f8-4d08-83c9-54bb33b65180" containerName="pull"
Oct 31 01:24:37.107556 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107514    5650 cpu_manager.go:414] "containerMap: removing container" podUID="6b1bcf70-2690-47c4-9ef4-c884b38c8bd8" containerName="pull"
Oct 31 01:24:37.107556 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107518    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b1bcf70-2690-47c4-9ef4-c884b38c8bd8" containerName="pull"
Oct 31 01:24:37.107556 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107524    5650 cpu_manager.go:414] "containerMap: removing container" podUID="0030819c-c3f8-4d08-83c9-54bb33b65180" containerName="extract"
Oct 31 01:24:37.107556 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107531    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0030819c-c3f8-4d08-83c9-54bb33b65180" containerName="extract"
Oct 31 01:24:37.107556 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107541    5650 cpu_manager.go:414] "containerMap: removing container" podUID="6b1bcf70-2690-47c4-9ef4-c884b38c8bd8" containerName="util"
Oct 31 01:24:37.107556 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107545    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b1bcf70-2690-47c4-9ef4-c884b38c8bd8" containerName="util"
Oct 31 01:24:37.107556 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107551    5650 cpu_manager.go:414] "containerMap: removing container" podUID="287df43e-6f58-4044-9fd7-88d240d327d0" containerName="extract"
Oct 31 01:24:37.107556 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107555    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="287df43e-6f58-4044-9fd7-88d240d327d0" containerName="extract"
Oct 31 01:24:37.107556 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107562    5650 cpu_manager.go:414] "containerMap: removing container" podUID="6b1bcf70-2690-47c4-9ef4-c884b38c8bd8" containerName="extract"
Oct 31 01:24:37.107963 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107567    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b1bcf70-2690-47c4-9ef4-c884b38c8bd8" containerName="extract"
Oct 31 01:24:37.107963 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107579    5650 cpu_manager.go:414] "containerMap: removing container" podUID="287df43e-6f58-4044-9fd7-88d240d327d0" containerName="pull"
Oct 31 01:24:37.107963 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107584    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="287df43e-6f58-4044-9fd7-88d240d327d0" containerName="pull"
Oct 31 01:24:37.107963 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107592    5650 cpu_manager.go:414] "containerMap: removing container" podUID="0030819c-c3f8-4d08-83c9-54bb33b65180" containerName="util"
Oct 31 01:24:37.107963 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107596    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0030819c-c3f8-4d08-83c9-54bb33b65180" containerName="util"
Oct 31 01:24:37.107963 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107602    5650 cpu_manager.go:414] "containerMap: removing container" podUID="41976930-1143-4b6e-9a1c-83c16cfc2d5a" containerName="extract"
Oct 31 01:24:37.107963 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107606    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41976930-1143-4b6e-9a1c-83c16cfc2d5a" containerName="extract"
Oct 31 01:24:37.107963 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107656    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="41976930-1143-4b6e-9a1c-83c16cfc2d5a" containerName="extract"
Oct 31 01:24:37.107963 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107668    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="0030819c-c3f8-4d08-83c9-54bb33b65180" containerName="extract"
Oct 31 01:24:37.107963 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107674    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="287df43e-6f58-4044-9fd7-88d240d327d0" containerName="extract"
Oct 31 01:24:37.107963 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.107680    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b1bcf70-2690-47c4-9ef4-c884b38c8bd8" containerName="extract"
Oct 31 01:24:37.109344 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.109328    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2sb2w"
Oct 31 01:24:37.116469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.116435    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 31 01:24:37.117703 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.117688    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-p6997\""
Oct 31 01:24:37.124221 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.124196    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2sb2w"]
Oct 31 01:24:37.146920 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.146897    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67"
Oct 31 01:24:37.147324 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.146897    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2mk67" event={"ID":"287df43e-6f58-4044-9fd7-88d240d327d0","Type":"ContainerDied","Data":"33e262a38dabbd2e1ee36a96ec9188d34d716cc319daf4a7b9a4abcb284932fc"}
Oct 31 01:24:37.147324 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.147009    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33e262a38dabbd2e1ee36a96ec9188d34d716cc319daf4a7b9a4abcb284932fc"
Oct 31 01:24:37.149197 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.149167    5650 generic.go:356] "Generic (PLEG): container finished" podID="2eca10cf-d21b-4f66-93c4-b8bb65733e65" containerID="15c6a8a4b8b5f8ab8a1902efe0e524f0a88b24a47bf1433aee769ebc2767c251" exitCode=0
Oct 31 01:24:37.149303 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.149213    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6" event={"ID":"2eca10cf-d21b-4f66-93c4-b8bb65733e65","Type":"ContainerDied","Data":"15c6a8a4b8b5f8ab8a1902efe0e524f0a88b24a47bf1433aee769ebc2767c251"}
Oct 31 01:24:37.216431 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.216406    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:24:37.229727 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.229542    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9086572e-c5fe-4074-bd04-30689cba7f94-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-2sb2w\" (UID: \"9086572e-c5fe-4074-bd04-30689cba7f94\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2sb2w"
Oct 31 01:24:37.229727 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.229596    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km5k6\" (UniqueName: \"kubernetes.io/projected/9086572e-c5fe-4074-bd04-30689cba7f94-kube-api-access-km5k6\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-2sb2w\" (UID: \"9086572e-c5fe-4074-bd04-30689cba7f94\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2sb2w"
Oct 31 01:24:37.273755 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.273731    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v"
Oct 31 01:24:37.330941 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.330847    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9086572e-c5fe-4074-bd04-30689cba7f94-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-2sb2w\" (UID: \"9086572e-c5fe-4074-bd04-30689cba7f94\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2sb2w"
Oct 31 01:24:37.330941 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.330902    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km5k6\" (UniqueName: \"kubernetes.io/projected/9086572e-c5fe-4074-bd04-30689cba7f94-kube-api-access-km5k6\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-2sb2w\" (UID: \"9086572e-c5fe-4074-bd04-30689cba7f94\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2sb2w"
Oct 31 01:24:37.331332 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.331298    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9086572e-c5fe-4074-bd04-30689cba7f94-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-2sb2w\" (UID: \"9086572e-c5fe-4074-bd04-30689cba7f94\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2sb2w"
Oct 31 01:24:37.345423 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.345386    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km5k6\" (UniqueName: \"kubernetes.io/projected/9086572e-c5fe-4074-bd04-30689cba7f94-kube-api-access-km5k6\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-2sb2w\" (UID: \"9086572e-c5fe-4074-bd04-30689cba7f94\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2sb2w"
Oct 31 01:24:37.419418 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.419330    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2sb2w"
Oct 31 01:24:37.431516 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.431470    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96d392e0-a2f1-49a8-8e4d-61513f6e2558-util\" (UniqueName: \"kubernetes.io/empty-dir/96d392e0-a2f1-49a8-8e4d-61513f6e2558-util\") pod \"96d392e0-a2f1-49a8-8e4d-61513f6e2558\" (UID: \"96d392e0-a2f1-49a8-8e4d-61513f6e2558\") "
Oct 31 01:24:37.431516 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.431517    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96d392e0-a2f1-49a8-8e4d-61513f6e2558-bundle\" (UniqueName: \"kubernetes.io/empty-dir/96d392e0-a2f1-49a8-8e4d-61513f6e2558-bundle\") pod \"96d392e0-a2f1-49a8-8e4d-61513f6e2558\" (UID: \"96d392e0-a2f1-49a8-8e4d-61513f6e2558\") "
Oct 31 01:24:37.431776 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.431563    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96d392e0-a2f1-49a8-8e4d-61513f6e2558-kube-api-access-njrwf\" (UniqueName: \"kubernetes.io/projected/96d392e0-a2f1-49a8-8e4d-61513f6e2558-kube-api-access-njrwf\") pod \"96d392e0-a2f1-49a8-8e4d-61513f6e2558\" (UID: \"96d392e0-a2f1-49a8-8e4d-61513f6e2558\") "
Oct 31 01:24:37.432371 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.432315    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d392e0-a2f1-49a8-8e4d-61513f6e2558-bundle" pod "96d392e0-a2f1-49a8-8e4d-61513f6e2558" (UID: "96d392e0-a2f1-49a8-8e4d-61513f6e2558"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:37.433961 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.433931    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d392e0-a2f1-49a8-8e4d-61513f6e2558-kube-api-access-njrwf" pod "96d392e0-a2f1-49a8-8e4d-61513f6e2558" (UID: "96d392e0-a2f1-49a8-8e4d-61513f6e2558"). InnerVolumeSpecName "kube-api-access-njrwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:37.435858 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.435833    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d392e0-a2f1-49a8-8e4d-61513f6e2558-util" pod "96d392e0-a2f1-49a8-8e4d-61513f6e2558" (UID: "96d392e0-a2f1-49a8-8e4d-61513f6e2558"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:37.532751 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.532716    5650 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96d392e0-a2f1-49a8-8e4d-61513f6e2558-util\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:37.532751 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.532747    5650 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96d392e0-a2f1-49a8-8e4d-61513f6e2558-bundle\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:37.532751 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.532757    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njrwf\" (UniqueName: \"kubernetes.io/projected/96d392e0-a2f1-49a8-8e4d-61513f6e2558-kube-api-access-njrwf\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:37.697568 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:37.697541    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2sb2w"]
Oct 31 01:24:37.699395 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:24:37.699366    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9086572e_c5fe_4074_bd04_30689cba7f94.slice/crio-f86f6b74cb0e5717949f6ec2dd0192a16c659ef3436ca27dfcd350bf266bff32 WatchSource:0}: Error finding container f86f6b74cb0e5717949f6ec2dd0192a16c659ef3436ca27dfcd350bf266bff32: Status 404 returned error can't find the container with id f86f6b74cb0e5717949f6ec2dd0192a16c659ef3436ca27dfcd350bf266bff32
Oct 31 01:24:38.155411 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:38.155383    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v"
Oct 31 01:24:38.155887 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:38.155406    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sh79v" event={"ID":"96d392e0-a2f1-49a8-8e4d-61513f6e2558","Type":"ContainerDied","Data":"8c9903b56e26da4c592fc480ba9e90313b0a122772000b9f5ad7fe5369b7295b"}
Oct 31 01:24:38.155887 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:38.155476    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c9903b56e26da4c592fc480ba9e90313b0a122772000b9f5ad7fe5369b7295b"
Oct 31 01:24:38.156996 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:38.156968    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2sb2w" event={"ID":"9086572e-c5fe-4074-bd04-30689cba7f94","Type":"ContainerStarted","Data":"f86f6b74cb0e5717949f6ec2dd0192a16c659ef3436ca27dfcd350bf266bff32"}
Oct 31 01:24:38.276409 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:38.276380    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6"
Oct 31 01:24:38.338628 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:38.338594    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eca10cf-d21b-4f66-93c4-b8bb65733e65-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2eca10cf-d21b-4f66-93c4-b8bb65733e65-bundle\") pod \"2eca10cf-d21b-4f66-93c4-b8bb65733e65\" (UID: \"2eca10cf-d21b-4f66-93c4-b8bb65733e65\") "
Oct 31 01:24:38.339140 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:38.339113    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eca10cf-d21b-4f66-93c4-b8bb65733e65-bundle" pod "2eca10cf-d21b-4f66-93c4-b8bb65733e65" (UID: "2eca10cf-d21b-4f66-93c4-b8bb65733e65"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:38.439402 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:38.439315    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eca10cf-d21b-4f66-93c4-b8bb65733e65-util\" (UniqueName: \"kubernetes.io/empty-dir/2eca10cf-d21b-4f66-93c4-b8bb65733e65-util\") pod \"2eca10cf-d21b-4f66-93c4-b8bb65733e65\" (UID: \"2eca10cf-d21b-4f66-93c4-b8bb65733e65\") "
Oct 31 01:24:38.439402 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:38.439372    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eca10cf-d21b-4f66-93c4-b8bb65733e65-kube-api-access-blkqm\" (UniqueName: \"kubernetes.io/projected/2eca10cf-d21b-4f66-93c4-b8bb65733e65-kube-api-access-blkqm\") pod \"2eca10cf-d21b-4f66-93c4-b8bb65733e65\" (UID: \"2eca10cf-d21b-4f66-93c4-b8bb65733e65\") "
Oct 31 01:24:38.439607 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:38.439590    5650 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2eca10cf-d21b-4f66-93c4-b8bb65733e65-bundle\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:38.441488 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:38.441441    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eca10cf-d21b-4f66-93c4-b8bb65733e65-kube-api-access-blkqm" pod "2eca10cf-d21b-4f66-93c4-b8bb65733e65" (UID: "2eca10cf-d21b-4f66-93c4-b8bb65733e65"). InnerVolumeSpecName "kube-api-access-blkqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:38.443622 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:38.443599    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eca10cf-d21b-4f66-93c4-b8bb65733e65-util" pod "2eca10cf-d21b-4f66-93c4-b8bb65733e65" (UID: "2eca10cf-d21b-4f66-93c4-b8bb65733e65"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:38.540011 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:38.539964    5650 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2eca10cf-d21b-4f66-93c4-b8bb65733e65-util\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:38.540011 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:38.540001    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blkqm\" (UniqueName: \"kubernetes.io/projected/2eca10cf-d21b-4f66-93c4-b8bb65733e65-kube-api-access-blkqm\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:39.165400 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:39.165364    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6" event={"ID":"2eca10cf-d21b-4f66-93c4-b8bb65733e65","Type":"ContainerDied","Data":"6f721da8f65871f5583d62ca64df0397f6d3fc891f16e2dc0d04fa54553d7829"}
Oct 31 01:24:39.165874 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:39.165427    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f721da8f65871f5583d62ca64df0397f6d3fc891f16e2dc0d04fa54553d7829"
Oct 31 01:24:39.165874 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:39.165543    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnrg6"
Oct 31 01:24:41.177069 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:41.177028    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2sb2w" event={"ID":"9086572e-c5fe-4074-bd04-30689cba7f94","Type":"ContainerStarted","Data":"1f68d73e4c5461066f9fe1d18245c8442a644f3eeb785f3d6d3db97b3c73a5ac"}
Oct 31 01:24:41.215164 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:41.215106    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2sb2w" podStartSLOduration=1.386417171 podStartE2EDuration="4.215090456s" totalImagesPullingTime="2.828673285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:41.2119188 +0000 UTC m=+237.215633092" watchObservedRunningTime="2026-10-31 01:24:41.215090456 +0000 UTC m=+237.218804748"
Oct 31 01:24:42.246081 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.246045    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz"]
Oct 31 01:24:42.246535 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.246343    5650 cpu_manager.go:414] "containerMap: removing container" podUID="2eca10cf-d21b-4f66-93c4-b8bb65733e65" containerName="util"
Oct 31 01:24:42.246535 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.246353    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eca10cf-d21b-4f66-93c4-b8bb65733e65" containerName="util"
Oct 31 01:24:42.246535 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.246367    5650 cpu_manager.go:414] "containerMap: removing container" podUID="96d392e0-a2f1-49a8-8e4d-61513f6e2558" containerName="util"
Oct 31 01:24:42.246535 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.246372    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96d392e0-a2f1-49a8-8e4d-61513f6e2558" containerName="util"
Oct 31 01:24:42.246535 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.246381    5650 cpu_manager.go:414] "containerMap: removing container" podUID="96d392e0-a2f1-49a8-8e4d-61513f6e2558" containerName="extract"
Oct 31 01:24:42.246535 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.246386    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96d392e0-a2f1-49a8-8e4d-61513f6e2558" containerName="extract"
Oct 31 01:24:42.246535 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.246406    5650 cpu_manager.go:414] "containerMap: removing container" podUID="2eca10cf-d21b-4f66-93c4-b8bb65733e65" containerName="extract"
Oct 31 01:24:42.246535 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.246411    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eca10cf-d21b-4f66-93c4-b8bb65733e65" containerName="extract"
Oct 31 01:24:42.246535 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.246417    5650 cpu_manager.go:414] "containerMap: removing container" podUID="96d392e0-a2f1-49a8-8e4d-61513f6e2558" containerName="pull"
Oct 31 01:24:42.246535 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.246422    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96d392e0-a2f1-49a8-8e4d-61513f6e2558" containerName="pull"
Oct 31 01:24:42.246535 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.246428    5650 cpu_manager.go:414] "containerMap: removing container" podUID="2eca10cf-d21b-4f66-93c4-b8bb65733e65" containerName="pull"
Oct 31 01:24:42.246535 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.246432    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eca10cf-d21b-4f66-93c4-b8bb65733e65" containerName="pull"
Oct 31 01:24:42.246535 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.246499    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="96d392e0-a2f1-49a8-8e4d-61513f6e2558" containerName="extract"
Oct 31 01:24:42.246535 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.246511    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eca10cf-d21b-4f66-93c4-b8bb65733e65" containerName="extract"
Oct 31 01:24:42.249615 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.249586    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz"
Oct 31 01:24:42.259699 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.259677    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 01:24:42.263138 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.263112    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 01:24:42.263138 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.263125    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 01:24:42.263286 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.263150    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 01:24:42.263286 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.263166    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-25fpn\""
Oct 31 01:24:42.263286 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.263236    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 01:24:42.312948 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.312913    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz"]
Oct 31 01:24:42.366150 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.366112    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9ghh\" (UniqueName: \"kubernetes.io/projected/f894ce10-c7d2-4f6d-bb37-cede5b9c92a6-kube-api-access-f9ghh\") pod \"opendatahub-operator-controller-manager-69496c6ffb-h8kpz\" (UID: \"f894ce10-c7d2-4f6d-bb37-cede5b9c92a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz"
Oct 31 01:24:42.366150 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.366153    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f894ce10-c7d2-4f6d-bb37-cede5b9c92a6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-h8kpz\" (UID: \"f894ce10-c7d2-4f6d-bb37-cede5b9c92a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz"
Oct 31 01:24:42.366367 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.366183    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f894ce10-c7d2-4f6d-bb37-cede5b9c92a6-manifests\") pod \"opendatahub-operator-controller-manager-69496c6ffb-h8kpz\" (UID: \"f894ce10-c7d2-4f6d-bb37-cede5b9c92a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz"
Oct 31 01:24:42.366367 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.366304    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f894ce10-c7d2-4f6d-bb37-cede5b9c92a6-webhook-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-h8kpz\" (UID: \"f894ce10-c7d2-4f6d-bb37-cede5b9c92a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz"
Oct 31 01:24:42.366367 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.366348    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f894ce10-c7d2-4f6d-bb37-cede5b9c92a6-metrics-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-h8kpz\" (UID: \"f894ce10-c7d2-4f6d-bb37-cede5b9c92a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz"
Oct 31 01:24:42.366506 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.366376    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f894ce10-c7d2-4f6d-bb37-cede5b9c92a6-tmp\") pod \"opendatahub-operator-controller-manager-69496c6ffb-h8kpz\" (UID: \"f894ce10-c7d2-4f6d-bb37-cede5b9c92a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz"
Oct 31 01:24:42.466809 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.466768    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f894ce10-c7d2-4f6d-bb37-cede5b9c92a6-manifests\") pod \"opendatahub-operator-controller-manager-69496c6ffb-h8kpz\" (UID: \"f894ce10-c7d2-4f6d-bb37-cede5b9c92a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz"
Oct 31 01:24:42.466986 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.466817    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f894ce10-c7d2-4f6d-bb37-cede5b9c92a6-webhook-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-h8kpz\" (UID: \"f894ce10-c7d2-4f6d-bb37-cede5b9c92a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz"
Oct 31 01:24:42.466986 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.466853    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f894ce10-c7d2-4f6d-bb37-cede5b9c92a6-metrics-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-h8kpz\" (UID: \"f894ce10-c7d2-4f6d-bb37-cede5b9c92a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz"
Oct 31 01:24:42.466986 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.466894    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f894ce10-c7d2-4f6d-bb37-cede5b9c92a6-tmp\") pod \"opendatahub-operator-controller-manager-69496c6ffb-h8kpz\" (UID: \"f894ce10-c7d2-4f6d-bb37-cede5b9c92a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz"
Oct 31 01:24:42.466986 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.466959    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9ghh\" (UniqueName: \"kubernetes.io/projected/f894ce10-c7d2-4f6d-bb37-cede5b9c92a6-kube-api-access-f9ghh\") pod \"opendatahub-operator-controller-manager-69496c6ffb-h8kpz\" (UID: \"f894ce10-c7d2-4f6d-bb37-cede5b9c92a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz"
Oct 31 01:24:42.467181 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.466997    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f894ce10-c7d2-4f6d-bb37-cede5b9c92a6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-h8kpz\" (UID: \"f894ce10-c7d2-4f6d-bb37-cede5b9c92a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz"
Oct 31 01:24:42.467276 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.467253    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f894ce10-c7d2-4f6d-bb37-cede5b9c92a6-manifests\") pod \"opendatahub-operator-controller-manager-69496c6ffb-h8kpz\" (UID: \"f894ce10-c7d2-4f6d-bb37-cede5b9c92a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz"
Oct 31 01:24:42.469537 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.469501    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f894ce10-c7d2-4f6d-bb37-cede5b9c92a6-tmp\") pod \"opendatahub-operator-controller-manager-69496c6ffb-h8kpz\" (UID: \"f894ce10-c7d2-4f6d-bb37-cede5b9c92a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz"
Oct 31 01:24:42.469671 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.469565    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f894ce10-c7d2-4f6d-bb37-cede5b9c92a6-metrics-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-h8kpz\" (UID: \"f894ce10-c7d2-4f6d-bb37-cede5b9c92a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz"
Oct 31 01:24:42.469671 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.469575    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f894ce10-c7d2-4f6d-bb37-cede5b9c92a6-webhook-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-h8kpz\" (UID: \"f894ce10-c7d2-4f6d-bb37-cede5b9c92a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz"
Oct 31 01:24:42.469747 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.469685    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f894ce10-c7d2-4f6d-bb37-cede5b9c92a6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-h8kpz\" (UID: \"f894ce10-c7d2-4f6d-bb37-cede5b9c92a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz"
Oct 31 01:24:42.502543 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.502468    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9ghh\" (UniqueName: \"kubernetes.io/projected/f894ce10-c7d2-4f6d-bb37-cede5b9c92a6-kube-api-access-f9ghh\") pod \"opendatahub-operator-controller-manager-69496c6ffb-h8kpz\" (UID: \"f894ce10-c7d2-4f6d-bb37-cede5b9c92a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz"
Oct 31 01:24:42.564394 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.564352    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz"
Oct 31 01:24:42.862726 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:24:42.862695    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf894ce10_c7d2_4f6d_bb37_cede5b9c92a6.slice/crio-e8ba6832fcf343dbf5e652e706148a1cd07f1974fe489d57b3636ad2eb8e8035 WatchSource:0}: Error finding container e8ba6832fcf343dbf5e652e706148a1cd07f1974fe489d57b3636ad2eb8e8035: Status 404 returned error can't find the container with id e8ba6832fcf343dbf5e652e706148a1cd07f1974fe489d57b3636ad2eb8e8035
Oct 31 01:24:42.863153 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:42.863131    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz"]
Oct 31 01:24:43.186710 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:43.186626    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz" event={"ID":"f894ce10-c7d2-4f6d-bb37-cede5b9c92a6","Type":"ContainerStarted","Data":"e8ba6832fcf343dbf5e652e706148a1cd07f1974fe489d57b3636ad2eb8e8035"}
Oct 31 01:24:45.106047 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:45.106012    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-trcrk"]
Oct 31 01:24:45.109301 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:45.109283    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-trcrk"
Oct 31 01:24:45.130507 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:45.130475    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-trcrk"]
Oct 31 01:24:45.131024 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:45.131003    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 31 01:24:45.131165 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:45.131058    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 31 01:24:45.131165 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:45.131140    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-6nvx7\""
Oct 31 01:24:45.192842 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:45.192806    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw9dm\" (UniqueName: \"kubernetes.io/projected/480a68fd-d09b-485c-a9a0-fd973e11076c-kube-api-access-pw9dm\") pod \"cert-manager-cainjector-5c57bbd4fc-trcrk\" (UID: \"480a68fd-d09b-485c-a9a0-fd973e11076c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-trcrk"
Oct 31 01:24:45.193034 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:45.192878    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/480a68fd-d09b-485c-a9a0-fd973e11076c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-trcrk\" (UID: \"480a68fd-d09b-485c-a9a0-fd973e11076c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-trcrk"
Oct 31 01:24:45.294079 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:45.294040    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw9dm\" (UniqueName: \"kubernetes.io/projected/480a68fd-d09b-485c-a9a0-fd973e11076c-kube-api-access-pw9dm\") pod \"cert-manager-cainjector-5c57bbd4fc-trcrk\" (UID: \"480a68fd-d09b-485c-a9a0-fd973e11076c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-trcrk"
Oct 31 01:24:45.294277 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:45.294219    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/480a68fd-d09b-485c-a9a0-fd973e11076c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-trcrk\" (UID: \"480a68fd-d09b-485c-a9a0-fd973e11076c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-trcrk"
Oct 31 01:24:45.307099 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:45.307073    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw9dm\" (UniqueName: \"kubernetes.io/projected/480a68fd-d09b-485c-a9a0-fd973e11076c-kube-api-access-pw9dm\") pod \"cert-manager-cainjector-5c57bbd4fc-trcrk\" (UID: \"480a68fd-d09b-485c-a9a0-fd973e11076c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-trcrk"
Oct 31 01:24:45.307225 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:45.307066    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/480a68fd-d09b-485c-a9a0-fd973e11076c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-trcrk\" (UID: \"480a68fd-d09b-485c-a9a0-fd973e11076c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-trcrk"
Oct 31 01:24:45.436636 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:45.436559    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-trcrk"
Oct 31 01:24:45.734361 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:45.734323    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-trcrk"]
Oct 31 01:24:45.738406 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:24:45.738367    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod480a68fd_d09b_485c_a9a0_fd973e11076c.slice/crio-aa19b4c5e02757ed0e11c2256beb493e13896d30f671d6487532df9fa48dc101 WatchSource:0}: Error finding container aa19b4c5e02757ed0e11c2256beb493e13896d30f671d6487532df9fa48dc101: Status 404 returned error can't find the container with id aa19b4c5e02757ed0e11c2256beb493e13896d30f671d6487532df9fa48dc101
Oct 31 01:24:46.202036 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:46.201947    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-trcrk" event={"ID":"480a68fd-d09b-485c-a9a0-fd973e11076c","Type":"ContainerStarted","Data":"aa19b4c5e02757ed0e11c2256beb493e13896d30f671d6487532df9fa48dc101"}
Oct 31 01:24:48.214009 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:48.213979    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz" event={"ID":"f894ce10-c7d2-4f6d-bb37-cede5b9c92a6","Type":"ContainerStarted","Data":"80ab8b455b60a43aed4ad97d5d5bb81b9a13294f38b9a7c95d3eb95cd974f894"}
Oct 31 01:24:48.589409 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:48.589338    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-gb22h"]
Oct 31 01:24:48.592717 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:48.592687    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gb22h"
Oct 31 01:24:48.602702 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:48.602663    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-2lkdl\""
Oct 31 01:24:48.602894 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:48.602871    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 31 01:24:48.604110 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:48.604089    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-gb22h"]
Oct 31 01:24:48.615056 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:48.615030    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:24:48.624927 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:48.624894    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97d71f76-3dc4-4215-894f-57b3dac21efb-tmp\") pod \"jobset-operator-674cd59dc5-gb22h\" (UID: \"97d71f76-3dc4-4215-894f-57b3dac21efb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gb22h"
Oct 31 01:24:48.625066 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:48.624958    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c7sz\" (UniqueName: \"kubernetes.io/projected/97d71f76-3dc4-4215-894f-57b3dac21efb-kube-api-access-9c7sz\") pod \"jobset-operator-674cd59dc5-gb22h\" (UID: \"97d71f76-3dc4-4215-894f-57b3dac21efb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gb22h"
Oct 31 01:24:48.726122 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:48.726087    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c7sz\" (UniqueName: \"kubernetes.io/projected/97d71f76-3dc4-4215-894f-57b3dac21efb-kube-api-access-9c7sz\") pod \"jobset-operator-674cd59dc5-gb22h\" (UID: \"97d71f76-3dc4-4215-894f-57b3dac21efb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gb22h"
Oct 31 01:24:48.726309 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:48.726188    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97d71f76-3dc4-4215-894f-57b3dac21efb-tmp\") pod \"jobset-operator-674cd59dc5-gb22h\" (UID: \"97d71f76-3dc4-4215-894f-57b3dac21efb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gb22h"
Oct 31 01:24:48.726667 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:48.726643    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97d71f76-3dc4-4215-894f-57b3dac21efb-tmp\") pod \"jobset-operator-674cd59dc5-gb22h\" (UID: \"97d71f76-3dc4-4215-894f-57b3dac21efb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gb22h"
Oct 31 01:24:48.756130 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:48.756100    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c7sz\" (UniqueName: \"kubernetes.io/projected/97d71f76-3dc4-4215-894f-57b3dac21efb-kube-api-access-9c7sz\") pod \"jobset-operator-674cd59dc5-gb22h\" (UID: \"97d71f76-3dc4-4215-894f-57b3dac21efb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gb22h"
Oct 31 01:24:48.907001 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:48.906906    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gb22h"
Oct 31 01:24:49.208608 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:49.208568    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-gb22h"]
Oct 31 01:24:49.211736 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:24:49.211702    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97d71f76_3dc4_4215_894f_57b3dac21efb.slice/crio-cdb95f76c16565739255608f998a8576767f5ab9f4e81d72d3caaaf83caf0d4b WatchSource:0}: Error finding container cdb95f76c16565739255608f998a8576767f5ab9f4e81d72d3caaaf83caf0d4b: Status 404 returned error can't find the container with id cdb95f76c16565739255608f998a8576767f5ab9f4e81d72d3caaaf83caf0d4b
Oct 31 01:24:49.221095 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:49.221061    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gb22h" event={"ID":"97d71f76-3dc4-4215-894f-57b3dac21efb","Type":"ContainerStarted","Data":"cdb95f76c16565739255608f998a8576767f5ab9f4e81d72d3caaaf83caf0d4b"}
Oct 31 01:24:49.222945 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:49.222917    5650 generic.go:356] "Generic (PLEG): container finished" podID="f894ce10-c7d2-4f6d-bb37-cede5b9c92a6" containerID="80ab8b455b60a43aed4ad97d5d5bb81b9a13294f38b9a7c95d3eb95cd974f894" exitCode=0
Oct 31 01:24:49.223075 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:49.222976    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz" event={"ID":"f894ce10-c7d2-4f6d-bb37-cede5b9c92a6","Type":"ContainerDied","Data":"80ab8b455b60a43aed4ad97d5d5bb81b9a13294f38b9a7c95d3eb95cd974f894"}
Oct 31 01:24:49.223075 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:49.223012    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz" event={"ID":"f894ce10-c7d2-4f6d-bb37-cede5b9c92a6","Type":"ContainerStarted","Data":"60b028d6316ee90e41263aee1b906b49fa1b5d33aa3241f127c26f31d71f0b27"}
Oct 31 01:24:49.223194 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:49.223088    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz"
Oct 31 01:24:49.288474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:49.288395    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz" podStartSLOduration=1.614797504 podStartE2EDuration="7.28837908s" totalImagesPullingTime="5.119850137s" totalInitContainerRuntime="553.731439ms" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:49.2849934 +0000 UTC m=+245.288707691" watchObservedRunningTime="2026-10-31 01:24:49.28837908 +0000 UTC m=+245.292093370"
Oct 31 01:24:49.540916 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:49.540836    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-28qvk"]
Oct 31 01:24:49.545505 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:49.545484    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-28qvk"
Oct 31 01:24:49.556065 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:49.556042    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-28qvk"]
Oct 31 01:24:49.558536 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:49.558512    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-qlwx9\""
Oct 31 01:24:49.635214 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:49.635176    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzh2d\" (UniqueName: \"kubernetes.io/projected/1548a9c1-776d-41f4-addb-399ab46f60c0-kube-api-access-gzh2d\") pod \"cert-manager-webhook-5dbbbf75cc-28qvk\" (UID: \"1548a9c1-776d-41f4-addb-399ab46f60c0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-28qvk"
Oct 31 01:24:49.635375 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:49.635242    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1548a9c1-776d-41f4-addb-399ab46f60c0-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-28qvk\" (UID: \"1548a9c1-776d-41f4-addb-399ab46f60c0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-28qvk"
Oct 31 01:24:49.736365 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:49.736325    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1548a9c1-776d-41f4-addb-399ab46f60c0-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-28qvk\" (UID: \"1548a9c1-776d-41f4-addb-399ab46f60c0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-28qvk"
Oct 31 01:24:49.736569 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:49.736502    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzh2d\" (UniqueName: \"kubernetes.io/projected/1548a9c1-776d-41f4-addb-399ab46f60c0-kube-api-access-gzh2d\") pod \"cert-manager-webhook-5dbbbf75cc-28qvk\" (UID: \"1548a9c1-776d-41f4-addb-399ab46f60c0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-28qvk"
Oct 31 01:24:49.750318 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:49.750283    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1548a9c1-776d-41f4-addb-399ab46f60c0-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-28qvk\" (UID: \"1548a9c1-776d-41f4-addb-399ab46f60c0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-28qvk"
Oct 31 01:24:49.750463 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:49.750414    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzh2d\" (UniqueName: \"kubernetes.io/projected/1548a9c1-776d-41f4-addb-399ab46f60c0-kube-api-access-gzh2d\") pod \"cert-manager-webhook-5dbbbf75cc-28qvk\" (UID: \"1548a9c1-776d-41f4-addb-399ab46f60c0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-28qvk"
Oct 31 01:24:49.858018 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:49.857931    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-28qvk"
Oct 31 01:24:50.150850 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:50.150751    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-28qvk"]
Oct 31 01:24:50.153163 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:24:50.153128    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1548a9c1_776d_41f4_addb_399ab46f60c0.slice/crio-4926e2201d14c08b8fc628874c84182a8552ea8a98461d68552b134fca61effb WatchSource:0}: Error finding container 4926e2201d14c08b8fc628874c84182a8552ea8a98461d68552b134fca61effb: Status 404 returned error can't find the container with id 4926e2201d14c08b8fc628874c84182a8552ea8a98461d68552b134fca61effb
Oct 31 01:24:50.228952 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:50.228905    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-28qvk" event={"ID":"1548a9c1-776d-41f4-addb-399ab46f60c0","Type":"ContainerStarted","Data":"4926e2201d14c08b8fc628874c84182a8552ea8a98461d68552b134fca61effb"}
Oct 31 01:24:52.243561 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:52.243513    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-trcrk" event={"ID":"480a68fd-d09b-485c-a9a0-fd973e11076c","Type":"ContainerStarted","Data":"a7bd26e6544eacbb432401ce845eac40cbc1015050f423523997526a070b1c4e"}
Oct 31 01:24:52.266545 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:52.266488    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-trcrk" podStartSLOduration=0.913920288 podStartE2EDuration="7.266471115s" totalImagesPullingTime="6.352550827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:52.264986372 +0000 UTC m=+248.268700663" watchObservedRunningTime="2026-10-31 01:24:52.266471115 +0000 UTC m=+248.270185403"
Oct 31 01:24:53.251886 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:53.251843    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-28qvk" event={"ID":"1548a9c1-776d-41f4-addb-399ab46f60c0","Type":"ContainerStarted","Data":"1155482e5c68f9b18a10be8423e0d96f83e0a065c87ec7ef001dba0313348a4d"}
Oct 31 01:24:53.274564 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:53.274497    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-28qvk" podStartSLOduration=2.199574063 podStartE2EDuration="4.274479155s" totalImagesPullingTime="2.074905092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:53.273748613 +0000 UTC m=+249.277462909" watchObservedRunningTime="2026-10-31 01:24:53.274479155 +0000 UTC m=+249.278193449"
Oct 31 01:24:54.258199 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:54.258157    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gb22h" event={"ID":"97d71f76-3dc4-4215-894f-57b3dac21efb","Type":"ContainerStarted","Data":"e5c73224207ccbbb370fe3cd52588a252ea4c86596ddb671a7131a50d91bbc4b"}
Oct 31 01:24:54.258694 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:54.258291    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-28qvk"
Oct 31 01:24:54.284900 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:24:54.284846    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gb22h" podStartSLOduration=2.123613905 podStartE2EDuration="6.284832564s" totalImagesPullingTime="4.161218659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:54.283831662 +0000 UTC m=+250.287545952" watchObservedRunningTime="2026-10-31 01:24:54.284832564 +0000 UTC m=+250.288546854"
Oct 31 01:24:58.551877 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:24:58.551840    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:25:00.231013 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:00.230978    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-h8kpz"
Oct 31 01:25:00.265362 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:00.265320    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-28qvk"
Oct 31 01:25:02.014206 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:02.014171    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-x7nxb"]
Oct 31 01:25:02.053497 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:02.053429    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-x7nxb"]
Oct 31 01:25:02.053664 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:02.053553    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-x7nxb"
Oct 31 01:25:02.073507 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:02.073474    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-rcqdl\""
Oct 31 01:25:02.150690 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:02.150652    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbe442b4-eff6-49ea-9cc2-6060a377c701-bound-sa-token\") pod \"cert-manager-597b95cb84-x7nxb\" (UID: \"fbe442b4-eff6-49ea-9cc2-6060a377c701\") " pod="cert-manager/cert-manager-597b95cb84-x7nxb"
Oct 31 01:25:02.150851 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:02.150697    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksn2w\" (UniqueName: \"kubernetes.io/projected/fbe442b4-eff6-49ea-9cc2-6060a377c701-kube-api-access-ksn2w\") pod \"cert-manager-597b95cb84-x7nxb\" (UID: \"fbe442b4-eff6-49ea-9cc2-6060a377c701\") " pod="cert-manager/cert-manager-597b95cb84-x7nxb"
Oct 31 01:25:02.251309 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:02.251269    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbe442b4-eff6-49ea-9cc2-6060a377c701-bound-sa-token\") pod \"cert-manager-597b95cb84-x7nxb\" (UID: \"fbe442b4-eff6-49ea-9cc2-6060a377c701\") " pod="cert-manager/cert-manager-597b95cb84-x7nxb"
Oct 31 01:25:02.251515 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:02.251324    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksn2w\" (UniqueName: \"kubernetes.io/projected/fbe442b4-eff6-49ea-9cc2-6060a377c701-kube-api-access-ksn2w\") pod \"cert-manager-597b95cb84-x7nxb\" (UID: \"fbe442b4-eff6-49ea-9cc2-6060a377c701\") " pod="cert-manager/cert-manager-597b95cb84-x7nxb"
Oct 31 01:25:02.265779 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:02.265706    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbe442b4-eff6-49ea-9cc2-6060a377c701-bound-sa-token\") pod \"cert-manager-597b95cb84-x7nxb\" (UID: \"fbe442b4-eff6-49ea-9cc2-6060a377c701\") " pod="cert-manager/cert-manager-597b95cb84-x7nxb"
Oct 31 01:25:02.265915 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:02.265893    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksn2w\" (UniqueName: \"kubernetes.io/projected/fbe442b4-eff6-49ea-9cc2-6060a377c701-kube-api-access-ksn2w\") pod \"cert-manager-597b95cb84-x7nxb\" (UID: \"fbe442b4-eff6-49ea-9cc2-6060a377c701\") " pod="cert-manager/cert-manager-597b95cb84-x7nxb"
Oct 31 01:25:02.380400 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:02.380359    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-x7nxb"
Oct 31 01:25:02.674324 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:02.674297    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-x7nxb"]
Oct 31 01:25:02.676908 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:25:02.676880    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbe442b4_eff6_49ea_9cc2_6060a377c701.slice/crio-ce4a578f490d0a0658fc382d7c1572a86515700620bde8dc6c093a7dbbe4f5f9 WatchSource:0}: Error finding container ce4a578f490d0a0658fc382d7c1572a86515700620bde8dc6c093a7dbbe4f5f9: Status 404 returned error can't find the container with id ce4a578f490d0a0658fc382d7c1572a86515700620bde8dc6c093a7dbbe4f5f9
Oct 31 01:25:03.303920 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:03.303881    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-x7nxb" event={"ID":"fbe442b4-eff6-49ea-9cc2-6060a377c701","Type":"ContainerStarted","Data":"ce610253ce07d2cfc7be963a21566dabf1f3f85b82cc52daa7491a446bb087ba"}
Oct 31 01:25:03.303920 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:03.303928    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-x7nxb" event={"ID":"fbe442b4-eff6-49ea-9cc2-6060a377c701","Type":"ContainerStarted","Data":"ce4a578f490d0a0658fc382d7c1572a86515700620bde8dc6c093a7dbbe4f5f9"}
Oct 31 01:25:03.370232 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:03.370161    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-x7nxb" podStartSLOduration=2.370145646 podStartE2EDuration="2.370145646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:03.368519131 +0000 UTC m=+259.372233419" watchObservedRunningTime="2026-10-31 01:25:03.370145646 +0000 UTC m=+259.373859937"
Oct 31 01:25:13.515139 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:13.515100    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6b44bd95d7-xxzfx"]
Oct 31 01:25:13.517935 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:13.517914    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-xxzfx"
Oct 31 01:25:13.542066 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:13.542032    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b44bd95d7-xxzfx"]
Oct 31 01:25:13.610178 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:13.610147    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 31 01:25:13.610178 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:13.610171    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 31 01:25:13.610427 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:13.610283    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jt94z\""
Oct 31 01:25:13.649352 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:13.649320    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac091ce7-c5c8-4c0a-936e-e3e7683c2841-tmp\") pod \"kube-auth-proxy-6b44bd95d7-xxzfx\" (UID: \"ac091ce7-c5c8-4c0a-936e-e3e7683c2841\") " pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-xxzfx"
Oct 31 01:25:13.649530 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:13.649362    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac091ce7-c5c8-4c0a-936e-e3e7683c2841-tls-certs\") pod \"kube-auth-proxy-6b44bd95d7-xxzfx\" (UID: \"ac091ce7-c5c8-4c0a-936e-e3e7683c2841\") " pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-xxzfx"
Oct 31 01:25:13.649530 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:13.649463    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn7vw\" (UniqueName: \"kubernetes.io/projected/ac091ce7-c5c8-4c0a-936e-e3e7683c2841-kube-api-access-xn7vw\") pod \"kube-auth-proxy-6b44bd95d7-xxzfx\" (UID: \"ac091ce7-c5c8-4c0a-936e-e3e7683c2841\") " pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-xxzfx"
Oct 31 01:25:13.750334 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:13.750295    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac091ce7-c5c8-4c0a-936e-e3e7683c2841-tmp\") pod \"kube-auth-proxy-6b44bd95d7-xxzfx\" (UID: \"ac091ce7-c5c8-4c0a-936e-e3e7683c2841\") " pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-xxzfx"
Oct 31 01:25:13.750334 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:13.750339    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac091ce7-c5c8-4c0a-936e-e3e7683c2841-tls-certs\") pod \"kube-auth-proxy-6b44bd95d7-xxzfx\" (UID: \"ac091ce7-c5c8-4c0a-936e-e3e7683c2841\") " pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-xxzfx"
Oct 31 01:25:13.750590 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:13.750366    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn7vw\" (UniqueName: \"kubernetes.io/projected/ac091ce7-c5c8-4c0a-936e-e3e7683c2841-kube-api-access-xn7vw\") pod \"kube-auth-proxy-6b44bd95d7-xxzfx\" (UID: \"ac091ce7-c5c8-4c0a-936e-e3e7683c2841\") " pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-xxzfx"
Oct 31 01:25:13.752758 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:13.752720    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac091ce7-c5c8-4c0a-936e-e3e7683c2841-tmp\") pod \"kube-auth-proxy-6b44bd95d7-xxzfx\" (UID: \"ac091ce7-c5c8-4c0a-936e-e3e7683c2841\") " pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-xxzfx"
Oct 31 01:25:13.752915 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:13.752897    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac091ce7-c5c8-4c0a-936e-e3e7683c2841-tls-certs\") pod \"kube-auth-proxy-6b44bd95d7-xxzfx\" (UID: \"ac091ce7-c5c8-4c0a-936e-e3e7683c2841\") " pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-xxzfx"
Oct 31 01:25:13.764367 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:13.764337    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn7vw\" (UniqueName: \"kubernetes.io/projected/ac091ce7-c5c8-4c0a-936e-e3e7683c2841-kube-api-access-xn7vw\") pod \"kube-auth-proxy-6b44bd95d7-xxzfx\" (UID: \"ac091ce7-c5c8-4c0a-936e-e3e7683c2841\") " pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-xxzfx"
Oct 31 01:25:13.829016 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:13.828931    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-xxzfx"
Oct 31 01:25:14.118285 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:14.118253    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b44bd95d7-xxzfx"]
Oct 31 01:25:14.120138 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:25:14.120114    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac091ce7_c5c8_4c0a_936e_e3e7683c2841.slice/crio-9bd7ca75ecbd4ee87f818efd85d69a09efd60b3a325b4d4694096f0da8a6005a WatchSource:0}: Error finding container 9bd7ca75ecbd4ee87f818efd85d69a09efd60b3a325b4d4694096f0da8a6005a: Status 404 returned error can't find the container with id 9bd7ca75ecbd4ee87f818efd85d69a09efd60b3a325b4d4694096f0da8a6005a
Oct 31 01:25:14.354414 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:14.354374    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-xxzfx" event={"ID":"ac091ce7-c5c8-4c0a-936e-e3e7683c2841","Type":"ContainerStarted","Data":"9bd7ca75ecbd4ee87f818efd85d69a09efd60b3a325b4d4694096f0da8a6005a"}
Oct 31 01:25:17.372265 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:17.372170    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-xxzfx" event={"ID":"ac091ce7-c5c8-4c0a-936e-e3e7683c2841","Type":"ContainerStarted","Data":"347e01f23ed44a60a21d09a103054b9e519fc654bdb65785f26238ae62c1fdd7"}
Oct 31 01:25:17.399078 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:17.399014    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6b44bd95d7-xxzfx" podStartSLOduration=1.39990822 podStartE2EDuration="4.398995694s" totalImagesPullingTime="2.999087474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:17.39797132 +0000 UTC m=+273.401685613" watchObservedRunningTime="2026-10-31 01:25:17.398995694 +0000 UTC m=+273.402709983"
Oct 31 01:25:23.144684 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:23.144649    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-769c9bf4d4-zvj6h"]
Oct 31 01:25:23.150568 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:23.150543    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-769c9bf4d4-zvj6h"
Oct 31 01:25:23.157629 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:23.157587    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-769c9bf4d4-zvj6h"]
Oct 31 01:25:23.158528 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:23.158499    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 31 01:25:23.158640 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:23.158560    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 31 01:25:23.158640 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:23.158565    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 31 01:25:23.158721 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:23.158651    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-mmdjr\""
Oct 31 01:25:23.339925 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:23.339882    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b4688ec-42f0-4806-8881-c9c02d301f63-metrics-certs\") pod \"jobset-controller-manager-769c9bf4d4-zvj6h\" (UID: \"0b4688ec-42f0-4806-8881-c9c02d301f63\") " pod="openshift-jobset-operator/jobset-controller-manager-769c9bf4d4-zvj6h"
Oct 31 01:25:23.339925 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:23.339930    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bs7f\" (UniqueName: \"kubernetes.io/projected/0b4688ec-42f0-4806-8881-c9c02d301f63-kube-api-access-7bs7f\") pod \"jobset-controller-manager-769c9bf4d4-zvj6h\" (UID: \"0b4688ec-42f0-4806-8881-c9c02d301f63\") " pod="openshift-jobset-operator/jobset-controller-manager-769c9bf4d4-zvj6h"
Oct 31 01:25:23.340174 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:23.340015    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b4688ec-42f0-4806-8881-c9c02d301f63-cert\") pod \"jobset-controller-manager-769c9bf4d4-zvj6h\" (UID: \"0b4688ec-42f0-4806-8881-c9c02d301f63\") " pod="openshift-jobset-operator/jobset-controller-manager-769c9bf4d4-zvj6h"
Oct 31 01:25:23.340174 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:23.340115    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0b4688ec-42f0-4806-8881-c9c02d301f63-manager-config\") pod \"jobset-controller-manager-769c9bf4d4-zvj6h\" (UID: \"0b4688ec-42f0-4806-8881-c9c02d301f63\") " pod="openshift-jobset-operator/jobset-controller-manager-769c9bf4d4-zvj6h"
Oct 31 01:25:23.441085 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:23.440990    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b4688ec-42f0-4806-8881-c9c02d301f63-metrics-certs\") pod \"jobset-controller-manager-769c9bf4d4-zvj6h\" (UID: \"0b4688ec-42f0-4806-8881-c9c02d301f63\") " pod="openshift-jobset-operator/jobset-controller-manager-769c9bf4d4-zvj6h"
Oct 31 01:25:23.441085 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:23.441042    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bs7f\" (UniqueName: \"kubernetes.io/projected/0b4688ec-42f0-4806-8881-c9c02d301f63-kube-api-access-7bs7f\") pod \"jobset-controller-manager-769c9bf4d4-zvj6h\" (UID: \"0b4688ec-42f0-4806-8881-c9c02d301f63\") " pod="openshift-jobset-operator/jobset-controller-manager-769c9bf4d4-zvj6h"
Oct 31 01:25:23.441327 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:23.441096    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b4688ec-42f0-4806-8881-c9c02d301f63-cert\") pod \"jobset-controller-manager-769c9bf4d4-zvj6h\" (UID: \"0b4688ec-42f0-4806-8881-c9c02d301f63\") " pod="openshift-jobset-operator/jobset-controller-manager-769c9bf4d4-zvj6h"
Oct 31 01:25:23.441327 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:23.441125    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0b4688ec-42f0-4806-8881-c9c02d301f63-manager-config\") pod \"jobset-controller-manager-769c9bf4d4-zvj6h\" (UID: \"0b4688ec-42f0-4806-8881-c9c02d301f63\") " pod="openshift-jobset-operator/jobset-controller-manager-769c9bf4d4-zvj6h"
Oct 31 01:25:23.441865 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:23.441840    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0b4688ec-42f0-4806-8881-c9c02d301f63-manager-config\") pod \"jobset-controller-manager-769c9bf4d4-zvj6h\" (UID: \"0b4688ec-42f0-4806-8881-c9c02d301f63\") " pod="openshift-jobset-operator/jobset-controller-manager-769c9bf4d4-zvj6h"
Oct 31 01:25:23.443602 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:23.443581    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b4688ec-42f0-4806-8881-c9c02d301f63-metrics-certs\") pod \"jobset-controller-manager-769c9bf4d4-zvj6h\" (UID: \"0b4688ec-42f0-4806-8881-c9c02d301f63\") " pod="openshift-jobset-operator/jobset-controller-manager-769c9bf4d4-zvj6h"
Oct 31 01:25:23.443714 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:23.443616    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b4688ec-42f0-4806-8881-c9c02d301f63-cert\") pod \"jobset-controller-manager-769c9bf4d4-zvj6h\" (UID: \"0b4688ec-42f0-4806-8881-c9c02d301f63\") " pod="openshift-jobset-operator/jobset-controller-manager-769c9bf4d4-zvj6h"
Oct 31 01:25:23.454758 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:23.454730    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bs7f\" (UniqueName: \"kubernetes.io/projected/0b4688ec-42f0-4806-8881-c9c02d301f63-kube-api-access-7bs7f\") pod \"jobset-controller-manager-769c9bf4d4-zvj6h\" (UID: \"0b4688ec-42f0-4806-8881-c9c02d301f63\") " pod="openshift-jobset-operator/jobset-controller-manager-769c9bf4d4-zvj6h"
Oct 31 01:25:23.463575 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:23.463550    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-769c9bf4d4-zvj6h"
Oct 31 01:25:23.736995 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:23.736967    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-769c9bf4d4-zvj6h"]
Oct 31 01:25:23.738930 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:25:23.738902    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b4688ec_42f0_4806_8881_c9c02d301f63.slice/crio-92c64929a6721ef79030de235cbc5120fff177b052e9a702cb95c2f7438f8aaf WatchSource:0}: Error finding container 92c64929a6721ef79030de235cbc5120fff177b052e9a702cb95c2f7438f8aaf: Status 404 returned error can't find the container with id 92c64929a6721ef79030de235cbc5120fff177b052e9a702cb95c2f7438f8aaf
Oct 31 01:25:24.406069 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:24.406030    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-769c9bf4d4-zvj6h" event={"ID":"0b4688ec-42f0-4806-8881-c9c02d301f63","Type":"ContainerStarted","Data":"92c64929a6721ef79030de235cbc5120fff177b052e9a702cb95c2f7438f8aaf"}
Oct 31 01:25:27.423893 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:27.423853    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-769c9bf4d4-zvj6h" event={"ID":"0b4688ec-42f0-4806-8881-c9c02d301f63","Type":"ContainerStarted","Data":"afc1b5df355eac334885d5367a6946071ca68a78d25a51d62650a06e9a43e602"}
Oct 31 01:25:27.424301 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:27.423939    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-769c9bf4d4-zvj6h"
Oct 31 01:25:27.446511 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:27.446460    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-769c9bf4d4-zvj6h" podStartSLOduration=1.507573775 podStartE2EDuration="4.446435035s" totalImagesPullingTime="2.93886126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:27.445889468 +0000 UTC m=+283.449603760" watchObservedRunningTime="2026-10-31 01:25:27.446435035 +0000 UTC m=+283.450149325"
Oct 31 01:25:28.552217 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:25:28.552178    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:25:38.434612 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:25:38.434573    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-769c9bf4d4-zvj6h"
Oct 31 01:25:58.551502 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:25:58.551466    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:26:28.551859 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:26:28.551823    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:26:58.552060 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:26:58.551981    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:27:28.551878 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:27:28.551831    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:27:58.551653 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:27:58.551609    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:28:28.551393 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:28:28.551309    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:28:58.551458 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:28:58.551410    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:29:28.551794 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:29:28.551758    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:29:58.551470 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:29:58.551369    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:30:28.552140 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:30:28.552100    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:30:58.551499 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:30:58.551457    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:31:28.552682 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:31:28.552596    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:31:58.552172 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:31:58.552133    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:32:28.552174 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:32:28.552136    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:32:58.551747 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:32:58.551663    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:33:28.550954 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:33:28.550912    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:33:58.551755 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:33:58.551715    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:34:28.551854 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:34:28.551779    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:34:58.552581 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:34:58.552475    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:35:28.550544 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:35:28.550499    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:35:52.810075 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:35:52.810038    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-w4tcj"]
Oct 31 01:35:52.813541 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:35:52.813522    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-w4tcj"
Oct 31 01:35:52.822104 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:35:52.822077    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-z2xhp\""
Oct 31 01:35:52.825334 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:35:52.825309    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-w4tcj"]
Oct 31 01:35:52.857545 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:35:52.857515    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 01:35:52.857699 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:35:52.857572    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 01:35:52.933057 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:35:52.933015    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndrdz\" (UniqueName: \"kubernetes.io/projected/95f4fb6d-3ce1-4617-b271-4c8847dee2d9-kube-api-access-ndrdz\") pod \"authorino-56c6c9ddd5-w4tcj\" (UID: \"95f4fb6d-3ce1-4617-b271-4c8847dee2d9\") " pod="kuadrant-system/authorino-56c6c9ddd5-w4tcj"
Oct 31 01:35:53.034313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:35:53.034280    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndrdz\" (UniqueName: \"kubernetes.io/projected/95f4fb6d-3ce1-4617-b271-4c8847dee2d9-kube-api-access-ndrdz\") pod \"authorino-56c6c9ddd5-w4tcj\" (UID: \"95f4fb6d-3ce1-4617-b271-4c8847dee2d9\") " pod="kuadrant-system/authorino-56c6c9ddd5-w4tcj"
Oct 31 01:35:53.047986 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:35:53.047950    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndrdz\" (UniqueName: \"kubernetes.io/projected/95f4fb6d-3ce1-4617-b271-4c8847dee2d9-kube-api-access-ndrdz\") pod \"authorino-56c6c9ddd5-w4tcj\" (UID: \"95f4fb6d-3ce1-4617-b271-4c8847dee2d9\") " pod="kuadrant-system/authorino-56c6c9ddd5-w4tcj"
Oct 31 01:35:53.126642 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:35:53.126563    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-w4tcj"
Oct 31 01:35:53.397621 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:35:53.397594    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-w4tcj"]
Oct 31 01:35:53.399323 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:35:53.399287    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95f4fb6d_3ce1_4617_b271_4c8847dee2d9.slice/crio-e39e0a00c12e15681056e6a1dfbdbb02864b092dfc0dcff91425ccde2d71765d WatchSource:0}: Error finding container e39e0a00c12e15681056e6a1dfbdbb02864b092dfc0dcff91425ccde2d71765d: Status 404 returned error can't find the container with id e39e0a00c12e15681056e6a1dfbdbb02864b092dfc0dcff91425ccde2d71765d
Oct 31 01:35:53.400649 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:35:53.400629    5650 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:35:54.308340 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:35:54.308304    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-w4tcj" event={"ID":"95f4fb6d-3ce1-4617-b271-4c8847dee2d9","Type":"ContainerStarted","Data":"e39e0a00c12e15681056e6a1dfbdbb02864b092dfc0dcff91425ccde2d71765d"}
Oct 31 01:35:58.553327 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:35:58.553233    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:36:00.339926 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:36:00.339838    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-w4tcj" event={"ID":"95f4fb6d-3ce1-4617-b271-4c8847dee2d9","Type":"ContainerStarted","Data":"1ff1c781f7647a685617317968b5b90dc19ee914b166a97177a6dbf8cd951a14"}
Oct 31 01:36:00.359553 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:36:00.359503    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-w4tcj" podStartSLOduration=1.788773553 podStartE2EDuration="8.359488178s" totalImagesPullingTime="6.570714625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:36:00.358640002 +0000 UTC m=+916.362354307" watchObservedRunningTime="2026-10-31 01:36:00.359488178 +0000 UTC m=+916.363202469"
Oct 31 01:36:28.551718 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:36:28.551676    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:36:58.551892 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:36:58.551854    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:37:21.559547 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:21.559512    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2lt7l"]
Oct 31 01:37:21.562975 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:21.562955    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-2lt7l"
Oct 31 01:37:21.572529 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:21.572494    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2lt7l"]
Oct 31 01:37:21.593769 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:21.593739    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2lt7l"]
Oct 31 01:37:21.593960 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:37:21.593940    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dqv5s], unattached volumes=[], failed to process volumes=[kube-api-access-dqv5s]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-2lt7l" podUID="d3a9eb70-33ae-4b40-bbca-e8c6d7a4972e"
Oct 31 01:37:21.672222 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:21.672184    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqv5s\" (UniqueName: \"kubernetes.io/projected/d3a9eb70-33ae-4b40-bbca-e8c6d7a4972e-kube-api-access-dqv5s\") pod \"authorino-6f6944bcdd-2lt7l\" (UID: \"d3a9eb70-33ae-4b40-bbca-e8c6d7a4972e\") " pod="kuadrant-system/authorino-6f6944bcdd-2lt7l"
Oct 31 01:37:21.722001 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:21.721962    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-2lt7l"
Oct 31 01:37:21.729090 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:21.729064    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-2lt7l"
Oct 31 01:37:21.773406 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:21.773366    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqv5s\" (UniqueName: \"kubernetes.io/projected/d3a9eb70-33ae-4b40-bbca-e8c6d7a4972e-kube-api-access-dqv5s\") pod \"authorino-6f6944bcdd-2lt7l\" (UID: \"d3a9eb70-33ae-4b40-bbca-e8c6d7a4972e\") " pod="kuadrant-system/authorino-6f6944bcdd-2lt7l"
Oct 31 01:37:21.787142 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:21.787107    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqv5s\" (UniqueName: \"kubernetes.io/projected/d3a9eb70-33ae-4b40-bbca-e8c6d7a4972e-kube-api-access-dqv5s\") pod \"authorino-6f6944bcdd-2lt7l\" (UID: \"d3a9eb70-33ae-4b40-bbca-e8c6d7a4972e\") " pod="kuadrant-system/authorino-6f6944bcdd-2lt7l"
Oct 31 01:37:21.874356 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:21.874266    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3a9eb70-33ae-4b40-bbca-e8c6d7a4972e-kube-api-access-dqv5s\" (UniqueName: \"kubernetes.io/projected/d3a9eb70-33ae-4b40-bbca-e8c6d7a4972e-kube-api-access-dqv5s\") pod \"d3a9eb70-33ae-4b40-bbca-e8c6d7a4972e\" (UID: \"d3a9eb70-33ae-4b40-bbca-e8c6d7a4972e\") "
Oct 31 01:37:21.876440 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:21.876408    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a9eb70-33ae-4b40-bbca-e8c6d7a4972e-kube-api-access-dqv5s" pod "d3a9eb70-33ae-4b40-bbca-e8c6d7a4972e" (UID: "d3a9eb70-33ae-4b40-bbca-e8c6d7a4972e"). InnerVolumeSpecName "kube-api-access-dqv5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:37:21.975642 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:21.975603    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqv5s\" (UniqueName: \"kubernetes.io/projected/d3a9eb70-33ae-4b40-bbca-e8c6d7a4972e-kube-api-access-dqv5s\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:37:22.003075 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:22.003040    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8bb865d4-fvkqf"]
Oct 31 01:37:22.006391 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:22.006374    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8bb865d4-fvkqf"
Oct 31 01:37:22.024183 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:22.024155    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 31 01:37:22.025645 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:22.025622    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8bb865d4-fvkqf"]
Oct 31 01:37:22.076374 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:22.076339    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnb6g\" (UniqueName: \"kubernetes.io/projected/9cd0556f-8541-4c36-80d8-8b18241d3ae8-kube-api-access-wnb6g\") pod \"authorino-8bb865d4-fvkqf\" (UID: \"9cd0556f-8541-4c36-80d8-8b18241d3ae8\") " pod="kuadrant-system/authorino-8bb865d4-fvkqf"
Oct 31 01:37:22.076549 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:22.076476    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9cd0556f-8541-4c36-80d8-8b18241d3ae8-tls-cert\") pod \"authorino-8bb865d4-fvkqf\" (UID: \"9cd0556f-8541-4c36-80d8-8b18241d3ae8\") " pod="kuadrant-system/authorino-8bb865d4-fvkqf"
Oct 31 01:37:22.177489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:22.177378    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9cd0556f-8541-4c36-80d8-8b18241d3ae8-tls-cert\") pod \"authorino-8bb865d4-fvkqf\" (UID: \"9cd0556f-8541-4c36-80d8-8b18241d3ae8\") " pod="kuadrant-system/authorino-8bb865d4-fvkqf"
Oct 31 01:37:22.177489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:22.177424    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnb6g\" (UniqueName: \"kubernetes.io/projected/9cd0556f-8541-4c36-80d8-8b18241d3ae8-kube-api-access-wnb6g\") pod \"authorino-8bb865d4-fvkqf\" (UID: \"9cd0556f-8541-4c36-80d8-8b18241d3ae8\") " pod="kuadrant-system/authorino-8bb865d4-fvkqf"
Oct 31 01:37:22.179850 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:22.179824    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9cd0556f-8541-4c36-80d8-8b18241d3ae8-tls-cert\") pod \"authorino-8bb865d4-fvkqf\" (UID: \"9cd0556f-8541-4c36-80d8-8b18241d3ae8\") " pod="kuadrant-system/authorino-8bb865d4-fvkqf"
Oct 31 01:37:22.191317 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:22.191290    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnb6g\" (UniqueName: \"kubernetes.io/projected/9cd0556f-8541-4c36-80d8-8b18241d3ae8-kube-api-access-wnb6g\") pod \"authorino-8bb865d4-fvkqf\" (UID: \"9cd0556f-8541-4c36-80d8-8b18241d3ae8\") " pod="kuadrant-system/authorino-8bb865d4-fvkqf"
Oct 31 01:37:22.317340 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:22.317297    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8bb865d4-fvkqf"
Oct 31 01:37:22.593369 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:22.593337    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8bb865d4-fvkqf"]
Oct 31 01:37:22.596104 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:37:22.596073    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cd0556f_8541_4c36_80d8_8b18241d3ae8.slice/crio-71794dd1d69d9ebdda1949e20a1f239a4006cf36b21b89cf2558bba06213612d WatchSource:0}: Error finding container 71794dd1d69d9ebdda1949e20a1f239a4006cf36b21b89cf2558bba06213612d: Status 404 returned error can't find the container with id 71794dd1d69d9ebdda1949e20a1f239a4006cf36b21b89cf2558bba06213612d
Oct 31 01:37:22.727552 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:22.727507    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8bb865d4-fvkqf" event={"ID":"9cd0556f-8541-4c36-80d8-8b18241d3ae8","Type":"ContainerStarted","Data":"71794dd1d69d9ebdda1949e20a1f239a4006cf36b21b89cf2558bba06213612d"}
Oct 31 01:37:22.727727 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:22.727592    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-2lt7l"
Oct 31 01:37:22.762699 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:22.762666    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2lt7l"]
Oct 31 01:37:22.767177 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:22.767153    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2lt7l"]
Oct 31 01:37:23.734247 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:23.734208    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8bb865d4-fvkqf" event={"ID":"9cd0556f-8541-4c36-80d8-8b18241d3ae8","Type":"ContainerStarted","Data":"69d8d0934ae3e8435649402dd59b98caa2b2d5d2981750f26b2261ef0fc79bc1"}
Oct 31 01:37:23.754020 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:23.753959    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-8bb865d4-fvkqf" podStartSLOduration=1.988023224 podStartE2EDuration="2.753945296s" totalImagesPullingTime="765.922072ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:37:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:37:23.753544374 +0000 UTC m=+999.757258678" watchObservedRunningTime="2026-10-31 01:37:23.753945296 +0000 UTC m=+999.757659586"
Oct 31 01:37:23.781474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:23.781409    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-w4tcj"]
Oct 31 01:37:23.781689 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:23.781662    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-w4tcj" podUID="95f4fb6d-3ce1-4617-b271-4c8847dee2d9" containerName="authorino" containerID="cri-o://1ff1c781f7647a685617317968b5b90dc19ee914b166a97177a6dbf8cd951a14" gracePeriod=30
Oct 31 01:37:24.049700 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:24.049672    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-w4tcj"
Oct 31 01:37:24.119159 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:24.119116    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3a9eb70-33ae-4b40-bbca-e8c6d7a4972e" path="/var/lib/kubelet/pods/d3a9eb70-33ae-4b40-bbca-e8c6d7a4972e/volumes"
Oct 31 01:37:24.203113 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:24.203071    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95f4fb6d-3ce1-4617-b271-4c8847dee2d9-kube-api-access-ndrdz\" (UniqueName: \"kubernetes.io/projected/95f4fb6d-3ce1-4617-b271-4c8847dee2d9-kube-api-access-ndrdz\") pod \"95f4fb6d-3ce1-4617-b271-4c8847dee2d9\" (UID: \"95f4fb6d-3ce1-4617-b271-4c8847dee2d9\") "
Oct 31 01:37:24.205353 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:24.205316    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f4fb6d-3ce1-4617-b271-4c8847dee2d9-kube-api-access-ndrdz" pod "95f4fb6d-3ce1-4617-b271-4c8847dee2d9" (UID: "95f4fb6d-3ce1-4617-b271-4c8847dee2d9"). InnerVolumeSpecName "kube-api-access-ndrdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:37:24.304218 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:24.304121    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndrdz\" (UniqueName: \"kubernetes.io/projected/95f4fb6d-3ce1-4617-b271-4c8847dee2d9-kube-api-access-ndrdz\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:37:24.740406 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:24.740371    5650 generic.go:356] "Generic (PLEG): container finished" podID="95f4fb6d-3ce1-4617-b271-4c8847dee2d9" containerID="1ff1c781f7647a685617317968b5b90dc19ee914b166a97177a6dbf8cd951a14" exitCode=0
Oct 31 01:37:24.740874 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:24.740440    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-w4tcj"
Oct 31 01:37:24.740874 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:24.740460    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-w4tcj" event={"ID":"95f4fb6d-3ce1-4617-b271-4c8847dee2d9","Type":"ContainerDied","Data":"1ff1c781f7647a685617317968b5b90dc19ee914b166a97177a6dbf8cd951a14"}
Oct 31 01:37:24.740874 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:24.740505    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-w4tcj" event={"ID":"95f4fb6d-3ce1-4617-b271-4c8847dee2d9","Type":"ContainerDied","Data":"e39e0a00c12e15681056e6a1dfbdbb02864b092dfc0dcff91425ccde2d71765d"}
Oct 31 01:37:24.740874 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:24.740522    5650 scope.go:122] "RemoveContainer" containerID="1ff1c781f7647a685617317968b5b90dc19ee914b166a97177a6dbf8cd951a14"
Oct 31 01:37:24.750013 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:24.749994    5650 scope.go:122] "RemoveContainer" containerID="1ff1c781f7647a685617317968b5b90dc19ee914b166a97177a6dbf8cd951a14"
Oct 31 01:37:24.750335 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:37:24.750314    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ff1c781f7647a685617317968b5b90dc19ee914b166a97177a6dbf8cd951a14\": container with ID starting with 1ff1c781f7647a685617317968b5b90dc19ee914b166a97177a6dbf8cd951a14 not found: ID does not exist" containerID="1ff1c781f7647a685617317968b5b90dc19ee914b166a97177a6dbf8cd951a14"
Oct 31 01:37:24.750389 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:24.750347    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff1c781f7647a685617317968b5b90dc19ee914b166a97177a6dbf8cd951a14"} err="failed to get container status \"1ff1c781f7647a685617317968b5b90dc19ee914b166a97177a6dbf8cd951a14\": rpc error: code = NotFound desc = could not find container \"1ff1c781f7647a685617317968b5b90dc19ee914b166a97177a6dbf8cd951a14\": container with ID starting with 1ff1c781f7647a685617317968b5b90dc19ee914b166a97177a6dbf8cd951a14 not found: ID does not exist"
Oct 31 01:37:24.764961 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:24.764930    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-w4tcj"]
Oct 31 01:37:24.769163 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:24.769137    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-w4tcj"]
Oct 31 01:37:26.116207 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:26.116170    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95f4fb6d-3ce1-4617-b271-4c8847dee2d9" path="/var/lib/kubelet/pods/95f4fb6d-3ce1-4617-b271-4c8847dee2d9/volumes"
Oct 31 01:37:28.551911 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:37:28.551841    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:37:58.552268 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:37:58.552230    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:37:59.124812 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:59.124776    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-4whxt"]
Oct 31 01:37:59.125109 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:59.125098    5650 cpu_manager.go:414] "containerMap: removing container" podUID="95f4fb6d-3ce1-4617-b271-4c8847dee2d9" containerName="authorino"
Oct 31 01:37:59.125162 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:59.125110    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95f4fb6d-3ce1-4617-b271-4c8847dee2d9" containerName="authorino"
Oct 31 01:37:59.125216 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:59.125207    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="95f4fb6d-3ce1-4617-b271-4c8847dee2d9" containerName="authorino"
Oct 31 01:37:59.129153 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:59.129136    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-4whxt"
Oct 31 01:37:59.139348 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:59.137669    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 31 01:37:59.139348 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:59.137681    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 31 01:37:59.139348 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:59.137694    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 01:37:59.139348 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:59.137697    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 01:37:59.139348 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:59.137772    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-cpwhl\""
Oct 31 01:37:59.139348 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:59.139169    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-4whxt"]
Oct 31 01:37:59.207083 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:59.207048    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vnf6\" (UniqueName: \"kubernetes.io/projected/bb5fa171-169f-47ff-b087-fe71542a20dd-kube-api-access-2vnf6\") pod \"ai-gateway-controller-7d85675f98-4whxt\" (UID: \"bb5fa171-169f-47ff-b087-fe71542a20dd\") " pod="opendatahub/ai-gateway-controller-7d85675f98-4whxt"
Oct 31 01:37:59.207252 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:59.207090    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5fa171-169f-47ff-b087-fe71542a20dd-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-4whxt\" (UID: \"bb5fa171-169f-47ff-b087-fe71542a20dd\") " pod="opendatahub/ai-gateway-controller-7d85675f98-4whxt"
Oct 31 01:37:59.307998 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:59.307962    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vnf6\" (UniqueName: \"kubernetes.io/projected/bb5fa171-169f-47ff-b087-fe71542a20dd-kube-api-access-2vnf6\") pod \"ai-gateway-controller-7d85675f98-4whxt\" (UID: \"bb5fa171-169f-47ff-b087-fe71542a20dd\") " pod="opendatahub/ai-gateway-controller-7d85675f98-4whxt"
Oct 31 01:37:59.307998 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:59.308002    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5fa171-169f-47ff-b087-fe71542a20dd-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-4whxt\" (UID: \"bb5fa171-169f-47ff-b087-fe71542a20dd\") " pod="opendatahub/ai-gateway-controller-7d85675f98-4whxt"
Oct 31 01:37:59.310463 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:59.310417    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5fa171-169f-47ff-b087-fe71542a20dd-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-4whxt\" (UID: \"bb5fa171-169f-47ff-b087-fe71542a20dd\") " pod="opendatahub/ai-gateway-controller-7d85675f98-4whxt"
Oct 31 01:37:59.321123 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:59.321091    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vnf6\" (UniqueName: \"kubernetes.io/projected/bb5fa171-169f-47ff-b087-fe71542a20dd-kube-api-access-2vnf6\") pod \"ai-gateway-controller-7d85675f98-4whxt\" (UID: \"bb5fa171-169f-47ff-b087-fe71542a20dd\") " pod="opendatahub/ai-gateway-controller-7d85675f98-4whxt"
Oct 31 01:37:59.443387 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:59.443282    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-4whxt"
Oct 31 01:37:59.717538 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:59.717483    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-4whxt"]
Oct 31 01:37:59.719895 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:37:59.719869    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5fa171_169f_47ff_b087_fe71542a20dd.slice/crio-cc0df6116cc9eaec28454ef270969b85d0f350c0e07d841fbeb52fe0614628f2 WatchSource:0}: Error finding container cc0df6116cc9eaec28454ef270969b85d0f350c0e07d841fbeb52fe0614628f2: Status 404 returned error can't find the container with id cc0df6116cc9eaec28454ef270969b85d0f350c0e07d841fbeb52fe0614628f2
Oct 31 01:37:59.911260 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:37:59.911217    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-4whxt" event={"ID":"bb5fa171-169f-47ff-b087-fe71542a20dd","Type":"ContainerStarted","Data":"cc0df6116cc9eaec28454ef270969b85d0f350c0e07d841fbeb52fe0614628f2"}
Oct 31 01:38:02.928722 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:02.928639    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-4whxt" event={"ID":"bb5fa171-169f-47ff-b087-fe71542a20dd","Type":"ContainerStarted","Data":"f0f18a8d604df5446d87c83dc69a5d8fcca9c83c77f6e2883fafea18554ed101"}
Oct 31 01:38:02.928722 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:02.928702    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-4whxt"
Oct 31 01:38:02.949711 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:02.949657    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-4whxt" podStartSLOduration=1.181374666 podStartE2EDuration="3.949644496s" totalImagesPullingTime="2.76826983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:37:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:02.949205262 +0000 UTC m=+1038.952919554" watchObservedRunningTime="2026-10-31 01:38:02.949644496 +0000 UTC m=+1038.953358788"
Oct 31 01:38:04.397072 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:04.397038    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-lq6ms"]
Oct 31 01:38:04.400623 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:04.400602    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-lq6ms"
Oct 31 01:38:04.409072 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:04.409045    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-lq6ms"]
Oct 31 01:38:04.453876 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:04.453831    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5f6f417-2dd1-4f4e-ad96-92d1f512cac1-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-lq6ms\" (UID: \"d5f6f417-2dd1-4f4e-ad96-92d1f512cac1\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-lq6ms"
Oct 31 01:38:04.454057 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:04.454012    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqz5j\" (UniqueName: \"kubernetes.io/projected/d5f6f417-2dd1-4f4e-ad96-92d1f512cac1-kube-api-access-cqz5j\") pod \"payload-processing-55fbbbf9dc-lq6ms\" (UID: \"d5f6f417-2dd1-4f4e-ad96-92d1f512cac1\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-lq6ms"
Oct 31 01:38:04.471506 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:04.471439    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 31 01:38:04.471506 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:04.471495    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-h5nk9\""
Oct 31 01:38:04.555504 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:04.555462    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqz5j\" (UniqueName: \"kubernetes.io/projected/d5f6f417-2dd1-4f4e-ad96-92d1f512cac1-kube-api-access-cqz5j\") pod \"payload-processing-55fbbbf9dc-lq6ms\" (UID: \"d5f6f417-2dd1-4f4e-ad96-92d1f512cac1\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-lq6ms"
Oct 31 01:38:04.555693 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:04.555540    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5f6f417-2dd1-4f4e-ad96-92d1f512cac1-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-lq6ms\" (UID: \"d5f6f417-2dd1-4f4e-ad96-92d1f512cac1\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-lq6ms"
Oct 31 01:38:04.556231 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:04.556209    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5f6f417-2dd1-4f4e-ad96-92d1f512cac1-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-lq6ms\" (UID: \"d5f6f417-2dd1-4f4e-ad96-92d1f512cac1\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-lq6ms"
Oct 31 01:38:04.568983 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:04.568954    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqz5j\" (UniqueName: \"kubernetes.io/projected/d5f6f417-2dd1-4f4e-ad96-92d1f512cac1-kube-api-access-cqz5j\") pod \"payload-processing-55fbbbf9dc-lq6ms\" (UID: \"d5f6f417-2dd1-4f4e-ad96-92d1f512cac1\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-lq6ms"
Oct 31 01:38:04.715668 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:04.715627    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-lq6ms"
Oct 31 01:38:04.987726 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:04.987700    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-lq6ms"]
Oct 31 01:38:04.989302 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:38:04.989272    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f6f417_2dd1_4f4e_ad96_92d1f512cac1.slice/crio-dac62c88c8b4b9bdbb62ac0fdadcf81441328c9bd5e629ee0053b8eeb3acd408 WatchSource:0}: Error finding container dac62c88c8b4b9bdbb62ac0fdadcf81441328c9bd5e629ee0053b8eeb3acd408: Status 404 returned error can't find the container with id dac62c88c8b4b9bdbb62ac0fdadcf81441328c9bd5e629ee0053b8eeb3acd408
Oct 31 01:38:05.945488 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:05.945433    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-lq6ms" event={"ID":"d5f6f417-2dd1-4f4e-ad96-92d1f512cac1","Type":"ContainerStarted","Data":"dac62c88c8b4b9bdbb62ac0fdadcf81441328c9bd5e629ee0053b8eeb3acd408"}
Oct 31 01:38:07.955672 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:07.955632    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-lq6ms" event={"ID":"d5f6f417-2dd1-4f4e-ad96-92d1f512cac1","Type":"ContainerStarted","Data":"45d435ac37101eb52c17bcda3ebb03cd6b654db96245b929b84ef1e3dc24b4fa"}
Oct 31 01:38:07.956078 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:07.955764    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-lq6ms"
Oct 31 01:38:07.977041 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:07.976989    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-lq6ms" podStartSLOduration=1.8055686469999999 podStartE2EDuration="3.97697633s" totalImagesPullingTime="2.171407683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:07.97647054 +0000 UTC m=+1043.980184828" watchObservedRunningTime="2026-10-31 01:38:07.97697633 +0000 UTC m=+1043.980690621"
Oct 31 01:38:13.936531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:13.936498    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-4whxt"
Oct 31 01:38:14.223807 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:14.223775    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-75686f85d5-6f7td"]
Oct 31 01:38:14.227422 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:14.227400    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-75686f85d5-6f7td"
Oct 31 01:38:14.236323 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:14.236292    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-75686f85d5-6f7td"]
Oct 31 01:38:14.345934 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:14.345893    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e714053-4cce-4192-a7a3-cc1ddb6c25fc-metrics-certs\") pod \"ai-gateway-controller-75686f85d5-6f7td\" (UID: \"4e714053-4cce-4192-a7a3-cc1ddb6c25fc\") " pod="opendatahub/ai-gateway-controller-75686f85d5-6f7td"
Oct 31 01:38:14.346118 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:14.345955    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdsls\" (UniqueName: \"kubernetes.io/projected/4e714053-4cce-4192-a7a3-cc1ddb6c25fc-kube-api-access-kdsls\") pod \"ai-gateway-controller-75686f85d5-6f7td\" (UID: \"4e714053-4cce-4192-a7a3-cc1ddb6c25fc\") " pod="opendatahub/ai-gateway-controller-75686f85d5-6f7td"
Oct 31 01:38:14.447167 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:14.447122    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e714053-4cce-4192-a7a3-cc1ddb6c25fc-metrics-certs\") pod \"ai-gateway-controller-75686f85d5-6f7td\" (UID: \"4e714053-4cce-4192-a7a3-cc1ddb6c25fc\") " pod="opendatahub/ai-gateway-controller-75686f85d5-6f7td"
Oct 31 01:38:14.447356 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:14.447196    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdsls\" (UniqueName: \"kubernetes.io/projected/4e714053-4cce-4192-a7a3-cc1ddb6c25fc-kube-api-access-kdsls\") pod \"ai-gateway-controller-75686f85d5-6f7td\" (UID: \"4e714053-4cce-4192-a7a3-cc1ddb6c25fc\") " pod="opendatahub/ai-gateway-controller-75686f85d5-6f7td"
Oct 31 01:38:14.449610 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:14.449586    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e714053-4cce-4192-a7a3-cc1ddb6c25fc-metrics-certs\") pod \"ai-gateway-controller-75686f85d5-6f7td\" (UID: \"4e714053-4cce-4192-a7a3-cc1ddb6c25fc\") " pod="opendatahub/ai-gateway-controller-75686f85d5-6f7td"
Oct 31 01:38:14.461046 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:14.461009    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdsls\" (UniqueName: \"kubernetes.io/projected/4e714053-4cce-4192-a7a3-cc1ddb6c25fc-kube-api-access-kdsls\") pod \"ai-gateway-controller-75686f85d5-6f7td\" (UID: \"4e714053-4cce-4192-a7a3-cc1ddb6c25fc\") " pod="opendatahub/ai-gateway-controller-75686f85d5-6f7td"
Oct 31 01:38:14.541412 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:14.541324    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-75686f85d5-6f7td"
Oct 31 01:38:14.818010 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:14.817982    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-75686f85d5-6f7td"]
Oct 31 01:38:14.819916 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:38:14.819884    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e714053_4cce_4192_a7a3_cc1ddb6c25fc.slice/crio-367ab92dbbbdac25efca04fe2568c8bda30851bea27bcae57faf225922ea3c06 WatchSource:0}: Error finding container 367ab92dbbbdac25efca04fe2568c8bda30851bea27bcae57faf225922ea3c06: Status 404 returned error can't find the container with id 367ab92dbbbdac25efca04fe2568c8bda30851bea27bcae57faf225922ea3c06
Oct 31 01:38:14.989269 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:14.989234    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-75686f85d5-6f7td" event={"ID":"4e714053-4cce-4192-a7a3-cc1ddb6c25fc","Type":"ContainerStarted","Data":"367ab92dbbbdac25efca04fe2568c8bda30851bea27bcae57faf225922ea3c06"}
Oct 31 01:38:15.995969 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:15.995932    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-75686f85d5-6f7td" event={"ID":"4e714053-4cce-4192-a7a3-cc1ddb6c25fc","Type":"ContainerStarted","Data":"714076cbd913d31c50dd82c50cb03bdf5abe65b1251b8d12fb1525834efb2c25"}
Oct 31 01:38:15.996402 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:15.996039    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-75686f85d5-6f7td"
Oct 31 01:38:16.019084 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:16.019033    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-75686f85d5-6f7td" podStartSLOduration=1.547095017 podStartE2EDuration="2.019019913s" totalImagesPullingTime="471.924896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:16.017715447 +0000 UTC m=+1052.021429740" watchObservedRunningTime="2026-10-31 01:38:16.019019913 +0000 UTC m=+1052.022734222"
Oct 31 01:38:18.964043 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:18.964010    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-lq6ms"
Oct 31 01:38:27.003517 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:27.003487    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-75686f85d5-6f7td"
Oct 31 01:38:27.054691 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:27.054651    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-4whxt"]
Oct 31 01:38:27.054943 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:27.054913    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-4whxt" podUID="bb5fa171-169f-47ff-b087-fe71542a20dd" containerName="manager" containerID="cri-o://f0f18a8d604df5446d87c83dc69a5d8fcca9c83c77f6e2883fafea18554ed101" gracePeriod=30
Oct 31 01:38:27.306466 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:27.306429    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-4whxt"
Oct 31 01:38:27.470359 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:27.470309    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5fa171-169f-47ff-b087-fe71542a20dd-kube-api-access-2vnf6\" (UniqueName: \"kubernetes.io/projected/bb5fa171-169f-47ff-b087-fe71542a20dd-kube-api-access-2vnf6\") pod \"bb5fa171-169f-47ff-b087-fe71542a20dd\" (UID: \"bb5fa171-169f-47ff-b087-fe71542a20dd\") "
Oct 31 01:38:27.470627 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:27.470500    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb5fa171-169f-47ff-b087-fe71542a20dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5fa171-169f-47ff-b087-fe71542a20dd-metrics-certs\") pod \"bb5fa171-169f-47ff-b087-fe71542a20dd\" (UID: \"bb5fa171-169f-47ff-b087-fe71542a20dd\") "
Oct 31 01:38:27.472941 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:27.472896    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb5fa171-169f-47ff-b087-fe71542a20dd-metrics-certs" pod "bb5fa171-169f-47ff-b087-fe71542a20dd" (UID: "bb5fa171-169f-47ff-b087-fe71542a20dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:38:27.473090 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:27.472943    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5fa171-169f-47ff-b087-fe71542a20dd-kube-api-access-2vnf6" pod "bb5fa171-169f-47ff-b087-fe71542a20dd" (UID: "bb5fa171-169f-47ff-b087-fe71542a20dd"). InnerVolumeSpecName "kube-api-access-2vnf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:38:27.571759 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:27.571659    5650 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5fa171-169f-47ff-b087-fe71542a20dd-metrics-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.571759 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:27.571700    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vnf6\" (UniqueName: \"kubernetes.io/projected/bb5fa171-169f-47ff-b087-fe71542a20dd-kube-api-access-2vnf6\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:38:28.055985 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:28.055949    5650 generic.go:356] "Generic (PLEG): container finished" podID="bb5fa171-169f-47ff-b087-fe71542a20dd" containerID="f0f18a8d604df5446d87c83dc69a5d8fcca9c83c77f6e2883fafea18554ed101" exitCode=0
Oct 31 01:38:28.056484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:28.056013    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-4whxt"
Oct 31 01:38:28.056484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:28.056021    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-4whxt" event={"ID":"bb5fa171-169f-47ff-b087-fe71542a20dd","Type":"ContainerDied","Data":"f0f18a8d604df5446d87c83dc69a5d8fcca9c83c77f6e2883fafea18554ed101"}
Oct 31 01:38:28.056484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:28.056090    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-4whxt" event={"ID":"bb5fa171-169f-47ff-b087-fe71542a20dd","Type":"ContainerDied","Data":"cc0df6116cc9eaec28454ef270969b85d0f350c0e07d841fbeb52fe0614628f2"}
Oct 31 01:38:28.056484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:28.056112    5650 scope.go:122] "RemoveContainer" containerID="f0f18a8d604df5446d87c83dc69a5d8fcca9c83c77f6e2883fafea18554ed101"
Oct 31 01:38:28.065596 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:28.065575    5650 scope.go:122] "RemoveContainer" containerID="f0f18a8d604df5446d87c83dc69a5d8fcca9c83c77f6e2883fafea18554ed101"
Oct 31 01:38:28.065915 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:38:28.065893    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f18a8d604df5446d87c83dc69a5d8fcca9c83c77f6e2883fafea18554ed101\": container with ID starting with f0f18a8d604df5446d87c83dc69a5d8fcca9c83c77f6e2883fafea18554ed101 not found: ID does not exist" containerID="f0f18a8d604df5446d87c83dc69a5d8fcca9c83c77f6e2883fafea18554ed101"
Oct 31 01:38:28.065955 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:28.065928    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f18a8d604df5446d87c83dc69a5d8fcca9c83c77f6e2883fafea18554ed101"} err="failed to get container status \"f0f18a8d604df5446d87c83dc69a5d8fcca9c83c77f6e2883fafea18554ed101\": rpc error: code = NotFound desc = could not find container \"f0f18a8d604df5446d87c83dc69a5d8fcca9c83c77f6e2883fafea18554ed101\": container with ID starting with f0f18a8d604df5446d87c83dc69a5d8fcca9c83c77f6e2883fafea18554ed101 not found: ID does not exist"
Oct 31 01:38:28.079858 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:28.079825    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-4whxt"]
Oct 31 01:38:28.084317 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:28.084282    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-4whxt"]
Oct 31 01:38:28.117167 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:28.117127    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb5fa171-169f-47ff-b087-fe71542a20dd" path="/var/lib/kubelet/pods/bb5fa171-169f-47ff-b087-fe71542a20dd/volumes"
Oct 31 01:38:28.552862 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:38:28.552816    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:38:36.020075 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.020044    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"]
Oct 31 01:38:36.021036 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.020415    5650 cpu_manager.go:414] "containerMap: removing container" podUID="bb5fa171-169f-47ff-b087-fe71542a20dd" containerName="manager"
Oct 31 01:38:36.021036 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.020425    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb5fa171-169f-47ff-b087-fe71542a20dd" containerName="manager"
Oct 31 01:38:36.021036 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.020521    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb5fa171-169f-47ff-b087-fe71542a20dd" containerName="manager"
Oct 31 01:38:36.026462 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.026431    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"
Oct 31 01:38:36.034223 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.034193    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"]
Oct 31 01:38:36.034328 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.034311    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 31 01:38:36.034370 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.034356    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 01:38:36.034535 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.034521    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 01:38:36.035231 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.035219    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bg6sl\""
Oct 31 01:38:36.041466 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.041430    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e31f257-42d1-4f16-9365-901371aa1356-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-g8s4b\" (UID: \"1e31f257-42d1-4f16-9365-901371aa1356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"
Oct 31 01:38:36.041571 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.041492    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj6zd\" (UniqueName: \"kubernetes.io/projected/1e31f257-42d1-4f16-9365-901371aa1356-kube-api-access-dj6zd\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-g8s4b\" (UID: \"1e31f257-42d1-4f16-9365-901371aa1356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"
Oct 31 01:38:36.041611 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.041561    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e31f257-42d1-4f16-9365-901371aa1356-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-g8s4b\" (UID: \"1e31f257-42d1-4f16-9365-901371aa1356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"
Oct 31 01:38:36.041611 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.041607    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e31f257-42d1-4f16-9365-901371aa1356-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-g8s4b\" (UID: \"1e31f257-42d1-4f16-9365-901371aa1356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"
Oct 31 01:38:36.041678 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.041664    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e31f257-42d1-4f16-9365-901371aa1356-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-g8s4b\" (UID: \"1e31f257-42d1-4f16-9365-901371aa1356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"
Oct 31 01:38:36.041711 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.041687    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e31f257-42d1-4f16-9365-901371aa1356-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-g8s4b\" (UID: \"1e31f257-42d1-4f16-9365-901371aa1356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"
Oct 31 01:38:36.142348 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.142308    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj6zd\" (UniqueName: \"kubernetes.io/projected/1e31f257-42d1-4f16-9365-901371aa1356-kube-api-access-dj6zd\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-g8s4b\" (UID: \"1e31f257-42d1-4f16-9365-901371aa1356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"
Oct 31 01:38:36.142570 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.142370    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e31f257-42d1-4f16-9365-901371aa1356-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-g8s4b\" (UID: \"1e31f257-42d1-4f16-9365-901371aa1356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"
Oct 31 01:38:36.142570 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.142415    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e31f257-42d1-4f16-9365-901371aa1356-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-g8s4b\" (UID: \"1e31f257-42d1-4f16-9365-901371aa1356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"
Oct 31 01:38:36.142570 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.142494    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e31f257-42d1-4f16-9365-901371aa1356-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-g8s4b\" (UID: \"1e31f257-42d1-4f16-9365-901371aa1356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"
Oct 31 01:38:36.142570 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.142519    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e31f257-42d1-4f16-9365-901371aa1356-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-g8s4b\" (UID: \"1e31f257-42d1-4f16-9365-901371aa1356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"
Oct 31 01:38:36.142796 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.142581    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e31f257-42d1-4f16-9365-901371aa1356-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-g8s4b\" (UID: \"1e31f257-42d1-4f16-9365-901371aa1356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"
Oct 31 01:38:36.142852 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.142829    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e31f257-42d1-4f16-9365-901371aa1356-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-g8s4b\" (UID: \"1e31f257-42d1-4f16-9365-901371aa1356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"
Oct 31 01:38:36.143023 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.142971    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e31f257-42d1-4f16-9365-901371aa1356-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-g8s4b\" (UID: \"1e31f257-42d1-4f16-9365-901371aa1356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"
Oct 31 01:38:36.143242 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.143209    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e31f257-42d1-4f16-9365-901371aa1356-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-g8s4b\" (UID: \"1e31f257-42d1-4f16-9365-901371aa1356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"
Oct 31 01:38:36.145014 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.144993    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e31f257-42d1-4f16-9365-901371aa1356-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-g8s4b\" (UID: \"1e31f257-42d1-4f16-9365-901371aa1356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"
Oct 31 01:38:36.145875 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.145859    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e31f257-42d1-4f16-9365-901371aa1356-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-g8s4b\" (UID: \"1e31f257-42d1-4f16-9365-901371aa1356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"
Oct 31 01:38:36.156539 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.156518    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj6zd\" (UniqueName: \"kubernetes.io/projected/1e31f257-42d1-4f16-9365-901371aa1356-kube-api-access-dj6zd\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-g8s4b\" (UID: \"1e31f257-42d1-4f16-9365-901371aa1356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"
Oct 31 01:38:36.340933 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.340834    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"
Oct 31 01:38:36.633400 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:38:36.631488    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e31f257_42d1_4f16_9365_901371aa1356.slice/crio-c897471a17085f780beffcb199fd86bfcfa8c3e7b6d2e2bbd4f37e8c500a28b2 WatchSource:0}: Error finding container c897471a17085f780beffcb199fd86bfcfa8c3e7b6d2e2bbd4f37e8c500a28b2: Status 404 returned error can't find the container with id c897471a17085f780beffcb199fd86bfcfa8c3e7b6d2e2bbd4f37e8c500a28b2
Oct 31 01:38:36.633400 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.633152    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"]
Oct 31 01:38:36.904334 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.904244    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"]
Oct 31 01:38:36.912505 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.910612    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"
Oct 31 01:38:36.914346 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.914310    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"]
Oct 31 01:38:36.920205 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.920102    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 31 01:38:36.948199 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.948161    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5321dae7-9deb-4010-99d4-2d9878bb62de-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rmmmc\" (UID: \"5321dae7-9deb-4010-99d4-2d9878bb62de\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"
Oct 31 01:38:36.948375 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.948257    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96h85\" (UniqueName: \"kubernetes.io/projected/5321dae7-9deb-4010-99d4-2d9878bb62de-kube-api-access-96h85\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rmmmc\" (UID: \"5321dae7-9deb-4010-99d4-2d9878bb62de\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"
Oct 31 01:38:36.948375 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.948319    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5321dae7-9deb-4010-99d4-2d9878bb62de-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rmmmc\" (UID: \"5321dae7-9deb-4010-99d4-2d9878bb62de\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"
Oct 31 01:38:36.948476 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.948374    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5321dae7-9deb-4010-99d4-2d9878bb62de-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rmmmc\" (UID: \"5321dae7-9deb-4010-99d4-2d9878bb62de\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"
Oct 31 01:38:36.948476 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.948456    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5321dae7-9deb-4010-99d4-2d9878bb62de-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rmmmc\" (UID: \"5321dae7-9deb-4010-99d4-2d9878bb62de\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"
Oct 31 01:38:36.948562 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:36.948488    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5321dae7-9deb-4010-99d4-2d9878bb62de-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rmmmc\" (UID: \"5321dae7-9deb-4010-99d4-2d9878bb62de\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"
Oct 31 01:38:37.049587 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:37.049211    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5321dae7-9deb-4010-99d4-2d9878bb62de-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rmmmc\" (UID: \"5321dae7-9deb-4010-99d4-2d9878bb62de\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"
Oct 31 01:38:37.049587 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:37.049267    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96h85\" (UniqueName: \"kubernetes.io/projected/5321dae7-9deb-4010-99d4-2d9878bb62de-kube-api-access-96h85\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rmmmc\" (UID: \"5321dae7-9deb-4010-99d4-2d9878bb62de\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"
Oct 31 01:38:37.049587 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:37.049299    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5321dae7-9deb-4010-99d4-2d9878bb62de-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rmmmc\" (UID: \"5321dae7-9deb-4010-99d4-2d9878bb62de\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"
Oct 31 01:38:37.049587 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:37.049332    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5321dae7-9deb-4010-99d4-2d9878bb62de-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rmmmc\" (UID: \"5321dae7-9deb-4010-99d4-2d9878bb62de\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"
Oct 31 01:38:37.049587 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:37.049367    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5321dae7-9deb-4010-99d4-2d9878bb62de-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rmmmc\" (UID: \"5321dae7-9deb-4010-99d4-2d9878bb62de\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"
Oct 31 01:38:37.049587 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:37.049395    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5321dae7-9deb-4010-99d4-2d9878bb62de-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rmmmc\" (UID: \"5321dae7-9deb-4010-99d4-2d9878bb62de\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"
Oct 31 01:38:37.052351 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:37.051912    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5321dae7-9deb-4010-99d4-2d9878bb62de-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rmmmc\" (UID: \"5321dae7-9deb-4010-99d4-2d9878bb62de\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"
Oct 31 01:38:37.052351 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:37.052101    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5321dae7-9deb-4010-99d4-2d9878bb62de-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rmmmc\" (UID: \"5321dae7-9deb-4010-99d4-2d9878bb62de\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"
Oct 31 01:38:37.052351 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:37.052316    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5321dae7-9deb-4010-99d4-2d9878bb62de-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rmmmc\" (UID: \"5321dae7-9deb-4010-99d4-2d9878bb62de\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"
Oct 31 01:38:37.053009 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:37.052965    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5321dae7-9deb-4010-99d4-2d9878bb62de-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rmmmc\" (UID: \"5321dae7-9deb-4010-99d4-2d9878bb62de\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"
Oct 31 01:38:37.054549 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:37.053942    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5321dae7-9deb-4010-99d4-2d9878bb62de-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rmmmc\" (UID: \"5321dae7-9deb-4010-99d4-2d9878bb62de\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"
Oct 31 01:38:37.066612 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:37.066585    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96h85\" (UniqueName: \"kubernetes.io/projected/5321dae7-9deb-4010-99d4-2d9878bb62de-kube-api-access-96h85\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rmmmc\" (UID: \"5321dae7-9deb-4010-99d4-2d9878bb62de\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"
Oct 31 01:38:37.114551 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:37.114518    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b" event={"ID":"1e31f257-42d1-4f16-9365-901371aa1356","Type":"ContainerStarted","Data":"c897471a17085f780beffcb199fd86bfcfa8c3e7b6d2e2bbd4f37e8c500a28b2"}
Oct 31 01:38:37.229962 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:37.229921    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"
Oct 31 01:38:37.528203 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:37.528162    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"]
Oct 31 01:38:37.531825 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:38:37.531782    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5321dae7_9deb_4010_99d4_2d9878bb62de.slice/crio-6a6b2bf1a0f94e1b40b4b3ef7c5d3d43832aacad12ba719fd29cbf6cdebf46df WatchSource:0}: Error finding container 6a6b2bf1a0f94e1b40b4b3ef7c5d3d43832aacad12ba719fd29cbf6cdebf46df: Status 404 returned error can't find the container with id 6a6b2bf1a0f94e1b40b4b3ef7c5d3d43832aacad12ba719fd29cbf6cdebf46df
Oct 31 01:38:38.123918 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:38.123880    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc" event={"ID":"5321dae7-9deb-4010-99d4-2d9878bb62de","Type":"ContainerStarted","Data":"6a6b2bf1a0f94e1b40b4b3ef7c5d3d43832aacad12ba719fd29cbf6cdebf46df"}
Oct 31 01:38:39.131209 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:39.131165    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b" event={"ID":"1e31f257-42d1-4f16-9365-901371aa1356","Type":"ContainerStarted","Data":"82bd849cb5af16c4b685dd7760e715b54bdbbfeec74187eb9cd9a867caf7e26b"}
Oct 31 01:38:39.132764 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:39.132738    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc" event={"ID":"5321dae7-9deb-4010-99d4-2d9878bb62de","Type":"ContainerStarted","Data":"412202c3f73c54aeb25a63b628991b58e6c0d8f1547aafc9c09ebaaeafe9f871"}
Oct 31 01:38:39.155664 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:39.155616    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b" podStartSLOduration=1.724757425 podStartE2EDuration="3.155602876s" totalImagesPullingTime="1.430845451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:39.15318914 +0000 UTC m=+1075.156903431" watchObservedRunningTime="2026-10-31 01:38:39.155602876 +0000 UTC m=+1075.159317167"
Oct 31 01:38:39.184179 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:39.184128    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc" podStartSLOduration=2.327998298 podStartE2EDuration="3.184113037s" totalImagesPullingTime="856.114739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:39.182490093 +0000 UTC m=+1075.186204386" watchObservedRunningTime="2026-10-31 01:38:39.184113037 +0000 UTC m=+1075.187827328"
Oct 31 01:38:39.231146 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:39.231108    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"
Oct 31 01:38:39.341540 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:39.341510    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"
Oct 31 01:38:40.610683 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:40.610649    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-rzqnw"]
Oct 31 01:38:40.618934 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:40.618909    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw"
Oct 31 01:38:40.625680 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:40.625646    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-rzqnw"]
Oct 31 01:38:40.626334 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:40.626300    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 31 01:38:40.680755 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:40.680712    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966529d0-0bf9-4d10-a2f3-035558a1c051-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-rzqnw\" (UID: \"966529d0-0bf9-4d10-a2f3-035558a1c051\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw"
Oct 31 01:38:40.680755 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:40.680754    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966529d0-0bf9-4d10-a2f3-035558a1c051-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-rzqnw\" (UID: \"966529d0-0bf9-4d10-a2f3-035558a1c051\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw"
Oct 31 01:38:40.680968 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:40.680773    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966529d0-0bf9-4d10-a2f3-035558a1c051-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-rzqnw\" (UID: \"966529d0-0bf9-4d10-a2f3-035558a1c051\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw"
Oct 31 01:38:40.680968 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:40.680934    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966529d0-0bf9-4d10-a2f3-035558a1c051-home\") pod \"echo-simulator-kserve-64b6f8f96c-rzqnw\" (UID: \"966529d0-0bf9-4d10-a2f3-035558a1c051\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw"
Oct 31 01:38:40.680968 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:40.680964    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966529d0-0bf9-4d10-a2f3-035558a1c051-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-rzqnw\" (UID: \"966529d0-0bf9-4d10-a2f3-035558a1c051\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw"
Oct 31 01:38:40.681111 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:40.680984    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9gsf\" (UniqueName: \"kubernetes.io/projected/966529d0-0bf9-4d10-a2f3-035558a1c051-kube-api-access-r9gsf\") pod \"echo-simulator-kserve-64b6f8f96c-rzqnw\" (UID: \"966529d0-0bf9-4d10-a2f3-035558a1c051\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw"
Oct 31 01:38:40.782184 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:40.782150    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966529d0-0bf9-4d10-a2f3-035558a1c051-home\") pod \"echo-simulator-kserve-64b6f8f96c-rzqnw\" (UID: \"966529d0-0bf9-4d10-a2f3-035558a1c051\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw"
Oct 31 01:38:40.782184 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:40.782187    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966529d0-0bf9-4d10-a2f3-035558a1c051-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-rzqnw\" (UID: \"966529d0-0bf9-4d10-a2f3-035558a1c051\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw"
Oct 31 01:38:40.782421 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:40.782206    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9gsf\" (UniqueName: \"kubernetes.io/projected/966529d0-0bf9-4d10-a2f3-035558a1c051-kube-api-access-r9gsf\") pod \"echo-simulator-kserve-64b6f8f96c-rzqnw\" (UID: \"966529d0-0bf9-4d10-a2f3-035558a1c051\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw"
Oct 31 01:38:40.782421 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:40.782286    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966529d0-0bf9-4d10-a2f3-035558a1c051-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-rzqnw\" (UID: \"966529d0-0bf9-4d10-a2f3-035558a1c051\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw"
Oct 31 01:38:40.782421 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:40.782312    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966529d0-0bf9-4d10-a2f3-035558a1c051-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-rzqnw\" (UID: \"966529d0-0bf9-4d10-a2f3-035558a1c051\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw"
Oct 31 01:38:40.782421 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:40.782340    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966529d0-0bf9-4d10-a2f3-035558a1c051-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-rzqnw\" (UID: \"966529d0-0bf9-4d10-a2f3-035558a1c051\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw"
Oct 31 01:38:40.782694 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:40.782667    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966529d0-0bf9-4d10-a2f3-035558a1c051-home\") pod \"echo-simulator-kserve-64b6f8f96c-rzqnw\" (UID: \"966529d0-0bf9-4d10-a2f3-035558a1c051\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw"
Oct 31 01:38:40.782694 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:40.782682    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966529d0-0bf9-4d10-a2f3-035558a1c051-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-rzqnw\" (UID: \"966529d0-0bf9-4d10-a2f3-035558a1c051\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw"
Oct 31 01:38:40.782810 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:40.782775    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966529d0-0bf9-4d10-a2f3-035558a1c051-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-rzqnw\" (UID: \"966529d0-0bf9-4d10-a2f3-035558a1c051\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw"
Oct 31 01:38:40.784665 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:40.784637    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966529d0-0bf9-4d10-a2f3-035558a1c051-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-rzqnw\" (UID: \"966529d0-0bf9-4d10-a2f3-035558a1c051\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw"
Oct 31 01:38:40.784788 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:40.784771    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966529d0-0bf9-4d10-a2f3-035558a1c051-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-rzqnw\" (UID: \"966529d0-0bf9-4d10-a2f3-035558a1c051\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw"
Oct 31 01:38:40.805185 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:40.805109    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9gsf\" (UniqueName: \"kubernetes.io/projected/966529d0-0bf9-4d10-a2f3-035558a1c051-kube-api-access-r9gsf\") pod \"echo-simulator-kserve-64b6f8f96c-rzqnw\" (UID: \"966529d0-0bf9-4d10-a2f3-035558a1c051\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw"
Oct 31 01:38:40.935945 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:40.935858    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw"
Oct 31 01:38:41.217062 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:41.217031    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-rzqnw"]
Oct 31 01:38:41.218392 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:38:41.218368    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966529d0_0bf9_4d10_a2f3_035558a1c051.slice/crio-131779adb49eaad5332f037b9c00ca5800e847a00c4ec9131836fcf2a5be588c WatchSource:0}: Error finding container 131779adb49eaad5332f037b9c00ca5800e847a00c4ec9131836fcf2a5be588c: Status 404 returned error can't find the container with id 131779adb49eaad5332f037b9c00ca5800e847a00c4ec9131836fcf2a5be588c
Oct 31 01:38:41.347564 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:41.347528    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"]
Oct 31 01:38:41.352798 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:41.352776    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"
Oct 31 01:38:41.365332 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:41.365299    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"]
Oct 31 01:38:41.366470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:41.366428    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 31 01:38:41.389024 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:41.388995    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7c22d93-6c25-479d-b877-15ed5e8ca249-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl\" (UID: \"c7c22d93-6c25-479d-b877-15ed5e8ca249\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"
Oct 31 01:38:41.389183 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:41.389031    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7c22d93-6c25-479d-b877-15ed5e8ca249-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl\" (UID: \"c7c22d93-6c25-479d-b877-15ed5e8ca249\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"
Oct 31 01:38:41.389183 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:41.389062    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7c22d93-6c25-479d-b877-15ed5e8ca249-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl\" (UID: \"c7c22d93-6c25-479d-b877-15ed5e8ca249\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"
Oct 31 01:38:41.389183 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:41.389142    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7c22d93-6c25-479d-b877-15ed5e8ca249-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl\" (UID: \"c7c22d93-6c25-479d-b877-15ed5e8ca249\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"
Oct 31 01:38:41.389297 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:41.389204    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7c22d93-6c25-479d-b877-15ed5e8ca249-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl\" (UID: \"c7c22d93-6c25-479d-b877-15ed5e8ca249\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"
Oct 31 01:38:41.389338 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:41.389326    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh7rp\" (UniqueName: \"kubernetes.io/projected/c7c22d93-6c25-479d-b877-15ed5e8ca249-kube-api-access-nh7rp\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl\" (UID: \"c7c22d93-6c25-479d-b877-15ed5e8ca249\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"
Oct 31 01:38:41.490196 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:41.490104    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh7rp\" (UniqueName: \"kubernetes.io/projected/c7c22d93-6c25-479d-b877-15ed5e8ca249-kube-api-access-nh7rp\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl\" (UID: \"c7c22d93-6c25-479d-b877-15ed5e8ca249\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"
Oct 31 01:38:41.490196 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:41.490151    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7c22d93-6c25-479d-b877-15ed5e8ca249-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl\" (UID: \"c7c22d93-6c25-479d-b877-15ed5e8ca249\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"
Oct 31 01:38:41.490196 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:41.490174    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7c22d93-6c25-479d-b877-15ed5e8ca249-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl\" (UID: \"c7c22d93-6c25-479d-b877-15ed5e8ca249\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"
Oct 31 01:38:41.490502 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:41.490201    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7c22d93-6c25-479d-b877-15ed5e8ca249-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl\" (UID: \"c7c22d93-6c25-479d-b877-15ed5e8ca249\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"
Oct 31 01:38:41.490502 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:41.490228    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7c22d93-6c25-479d-b877-15ed5e8ca249-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl\" (UID: \"c7c22d93-6c25-479d-b877-15ed5e8ca249\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"
Oct 31 01:38:41.490502 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:41.490251    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7c22d93-6c25-479d-b877-15ed5e8ca249-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl\" (UID: \"c7c22d93-6c25-479d-b877-15ed5e8ca249\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"
Oct 31 01:38:41.490740 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:41.490708    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7c22d93-6c25-479d-b877-15ed5e8ca249-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl\" (UID: \"c7c22d93-6c25-479d-b877-15ed5e8ca249\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"
Oct 31 01:38:41.490740 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:41.490724    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7c22d93-6c25-479d-b877-15ed5e8ca249-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl\" (UID: \"c7c22d93-6c25-479d-b877-15ed5e8ca249\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"
Oct 31 01:38:41.490928 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:41.490779    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7c22d93-6c25-479d-b877-15ed5e8ca249-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl\" (UID: \"c7c22d93-6c25-479d-b877-15ed5e8ca249\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"
Oct 31 01:38:41.492546 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:41.492522    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7c22d93-6c25-479d-b877-15ed5e8ca249-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl\" (UID: \"c7c22d93-6c25-479d-b877-15ed5e8ca249\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"
Oct 31 01:38:41.492732 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:41.492703    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7c22d93-6c25-479d-b877-15ed5e8ca249-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl\" (UID: \"c7c22d93-6c25-479d-b877-15ed5e8ca249\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"
Oct 31 01:38:41.504175 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:41.504146    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh7rp\" (UniqueName: \"kubernetes.io/projected/c7c22d93-6c25-479d-b877-15ed5e8ca249-kube-api-access-nh7rp\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl\" (UID: \"c7c22d93-6c25-479d-b877-15ed5e8ca249\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"
Oct 31 01:38:41.678120 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:41.678081    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"
Oct 31 01:38:41.960427 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:41.960397    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"]
Oct 31 01:38:41.962132 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:38:41.962104    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c22d93_6c25_479d_b877_15ed5e8ca249.slice/crio-45a70ce57d4ee71d07a27b6a2d0335918ddf9e3d3a1c86ca9c554daba6126639 WatchSource:0}: Error finding container 45a70ce57d4ee71d07a27b6a2d0335918ddf9e3d3a1c86ca9c554daba6126639: Status 404 returned error can't find the container with id 45a70ce57d4ee71d07a27b6a2d0335918ddf9e3d3a1c86ca9c554daba6126639
Oct 31 01:38:42.150710 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:42.150668    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw" event={"ID":"966529d0-0bf9-4d10-a2f3-035558a1c051","Type":"ContainerStarted","Data":"131779adb49eaad5332f037b9c00ca5800e847a00c4ec9131836fcf2a5be588c"}
Oct 31 01:38:42.152052 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:42.152021    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl" event={"ID":"c7c22d93-6c25-479d-b877-15ed5e8ca249","Type":"ContainerStarted","Data":"45a70ce57d4ee71d07a27b6a2d0335918ddf9e3d3a1c86ca9c554daba6126639"}
Oct 31 01:38:43.159870 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:43.159836    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl" event={"ID":"c7c22d93-6c25-479d-b877-15ed5e8ca249","Type":"ContainerStarted","Data":"13308a1b7d5d4e13fa7b4bc992d6a419151de20d81c21d516998585d26b1c690"}
Oct 31 01:38:43.182594 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:43.182538    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl" podStartSLOduration=1.856867762 podStartE2EDuration="2.18252416s" totalImagesPullingTime="325.656398ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:43.181520416 +0000 UTC m=+1079.185234707" watchObservedRunningTime="2026-10-31 01:38:43.18252416 +0000 UTC m=+1079.186238451"
Oct 31 01:38:43.678979 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:43.678885    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"
Oct 31 01:38:44.167618 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:44.167582    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw" event={"ID":"966529d0-0bf9-4d10-a2f3-035558a1c051","Type":"ContainerStarted","Data":"6ee120055142a504528667dc5dac7e5a9d3075f436e7aca82afdd319cc69b810"}
Oct 31 01:38:44.194341 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:44.194290    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw" podStartSLOduration=2.045685313 podStartE2EDuration="4.194274956s" totalImagesPullingTime="2.148589643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:44.191681833 +0000 UTC m=+1080.195396128" watchObservedRunningTime="2026-10-31 01:38:44.194274956 +0000 UTC m=+1080.197989267"
Oct 31 01:38:44.936903 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:44.936867    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw"
Oct 31 01:38:46.341609 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:46.341573    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"
Oct 31 01:38:46.353679 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:46.353654    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"
Oct 31 01:38:47.195005 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:47.194975    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-g8s4b"
Oct 31 01:38:47.231210 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:47.231169    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"
Oct 31 01:38:47.248492 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:47.248457    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"
Oct 31 01:38:48.200921 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:48.200888    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rmmmc"
Oct 31 01:38:50.936854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:50.936817    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw"
Oct 31 01:38:50.949083 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:50.949052    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw"
Oct 31 01:38:51.215517 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:51.215487    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-rzqnw"
Oct 31 01:38:51.678600 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:51.678509    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"
Oct 31 01:38:51.690819 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:51.690788    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"
Oct 31 01:38:52.221185 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:52.221148    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl"
Oct 31 01:38:56.864560 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:56.864475    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-544f7c9987-fwhzc"]
Oct 31 01:38:56.869341 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:56.869318    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-544f7c9987-fwhzc"
Oct 31 01:38:56.875663 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:56.875627    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-544f7c9987-fwhzc"]
Oct 31 01:38:56.943338 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:56.943300    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w67r8\" (UniqueName: \"kubernetes.io/projected/f0f46d02-6b50-40ec-b940-f767a9ce2b72-kube-api-access-w67r8\") pod \"authorino-544f7c9987-fwhzc\" (UID: \"f0f46d02-6b50-40ec-b940-f767a9ce2b72\") " pod="kuadrant-system/authorino-544f7c9987-fwhzc"
Oct 31 01:38:56.943536 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:56.943374    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0f46d02-6b50-40ec-b940-f767a9ce2b72-tls-cert\") pod \"authorino-544f7c9987-fwhzc\" (UID: \"f0f46d02-6b50-40ec-b940-f767a9ce2b72\") " pod="kuadrant-system/authorino-544f7c9987-fwhzc"
Oct 31 01:38:57.044307 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:57.044244    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w67r8\" (UniqueName: \"kubernetes.io/projected/f0f46d02-6b50-40ec-b940-f767a9ce2b72-kube-api-access-w67r8\") pod \"authorino-544f7c9987-fwhzc\" (UID: \"f0f46d02-6b50-40ec-b940-f767a9ce2b72\") " pod="kuadrant-system/authorino-544f7c9987-fwhzc"
Oct 31 01:38:57.044484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:57.044343    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0f46d02-6b50-40ec-b940-f767a9ce2b72-tls-cert\") pod \"authorino-544f7c9987-fwhzc\" (UID: \"f0f46d02-6b50-40ec-b940-f767a9ce2b72\") " pod="kuadrant-system/authorino-544f7c9987-fwhzc"
Oct 31 01:38:57.046881 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:57.046821    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f0f46d02-6b50-40ec-b940-f767a9ce2b72-tls-cert\") pod \"authorino-544f7c9987-fwhzc\" (UID: \"f0f46d02-6b50-40ec-b940-f767a9ce2b72\") " pod="kuadrant-system/authorino-544f7c9987-fwhzc"
Oct 31 01:38:57.058235 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:57.058208    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w67r8\" (UniqueName: \"kubernetes.io/projected/f0f46d02-6b50-40ec-b940-f767a9ce2b72-kube-api-access-w67r8\") pod \"authorino-544f7c9987-fwhzc\" (UID: \"f0f46d02-6b50-40ec-b940-f767a9ce2b72\") " pod="kuadrant-system/authorino-544f7c9987-fwhzc"
Oct 31 01:38:57.183031 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:57.182944    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-544f7c9987-fwhzc"
Oct 31 01:38:57.459232 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:57.459207    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-544f7c9987-fwhzc"]
Oct 31 01:38:57.461135 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:38:57.461098    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0f46d02_6b50_40ec_b940_f767a9ce2b72.slice/crio-fd45821b3534d58aa7178890668b471bb6b40f27ab24055210c41cb5d77a5a97 WatchSource:0}: Error finding container fd45821b3534d58aa7178890668b471bb6b40f27ab24055210c41cb5d77a5a97: Status 404 returned error can't find the container with id fd45821b3534d58aa7178890668b471bb6b40f27ab24055210c41cb5d77a5a97
Oct 31 01:38:58.243822 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:58.243781    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-544f7c9987-fwhzc" event={"ID":"f0f46d02-6b50-40ec-b940-f767a9ce2b72","Type":"ContainerStarted","Data":"fd45821b3534d58aa7178890668b471bb6b40f27ab24055210c41cb5d77a5a97"}
Oct 31 01:38:58.557075 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:38:58.557031    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:38:59.256094 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.256044    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-544f7c9987-fwhzc" event={"ID":"f0f46d02-6b50-40ec-b940-f767a9ce2b72","Type":"ContainerStarted","Data":"515ce89b024c5392c4125cac7359c48dc3fc164e2c9a7ce2f5bf8f43c3cda176"}
Oct 31 01:38:59.278470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.277405    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-544f7c9987-fwhzc" podStartSLOduration=2.219934975 podStartE2EDuration="3.277384422s" totalImagesPullingTime="1.057449447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:59.276378885 +0000 UTC m=+1095.280093177" watchObservedRunningTime="2026-10-31 01:38:59.277384422 +0000 UTC m=+1095.281098715"
Oct 31 01:38:59.311215 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.311172    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8bb865d4-fvkqf"]
Oct 31 01:38:59.311470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.311423    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-8bb865d4-fvkqf" podUID="9cd0556f-8541-4c36-80d8-8b18241d3ae8" containerName="authorino" containerID="cri-o://69d8d0934ae3e8435649402dd59b98caa2b2d5d2981750f26b2261ef0fc79bc1" gracePeriod=30
Oct 31 01:38:59.607093 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.607058    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8bb865d4-fvkqf"
Oct 31 01:38:59.673227 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.673192    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd0556f-8541-4c36-80d8-8b18241d3ae8-kube-api-access-wnb6g\" (UniqueName: \"kubernetes.io/projected/9cd0556f-8541-4c36-80d8-8b18241d3ae8-kube-api-access-wnb6g\") pod \"9cd0556f-8541-4c36-80d8-8b18241d3ae8\" (UID: \"9cd0556f-8541-4c36-80d8-8b18241d3ae8\") "
Oct 31 01:38:59.673391 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.673296    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cd0556f-8541-4c36-80d8-8b18241d3ae8-tls-cert\" (UniqueName: \"kubernetes.io/secret/9cd0556f-8541-4c36-80d8-8b18241d3ae8-tls-cert\") pod \"9cd0556f-8541-4c36-80d8-8b18241d3ae8\" (UID: \"9cd0556f-8541-4c36-80d8-8b18241d3ae8\") "
Oct 31 01:38:59.675480 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.675420    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd0556f-8541-4c36-80d8-8b18241d3ae8-kube-api-access-wnb6g" pod "9cd0556f-8541-4c36-80d8-8b18241d3ae8" (UID: "9cd0556f-8541-4c36-80d8-8b18241d3ae8"). InnerVolumeSpecName "kube-api-access-wnb6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:38:59.684836 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.684809    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd0556f-8541-4c36-80d8-8b18241d3ae8-tls-cert" pod "9cd0556f-8541-4c36-80d8-8b18241d3ae8" (UID: "9cd0556f-8541-4c36-80d8-8b18241d3ae8"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:38:59.714191 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.714152    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"]
Oct 31 01:38:59.714630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.714611    5650 cpu_manager.go:414] "containerMap: removing container" podUID="9cd0556f-8541-4c36-80d8-8b18241d3ae8" containerName="authorino"
Oct 31 01:38:59.714630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.714627    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cd0556f-8541-4c36-80d8-8b18241d3ae8" containerName="authorino"
Oct 31 01:38:59.714809 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.714688    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cd0556f-8541-4c36-80d8-8b18241d3ae8" containerName="authorino"
Oct 31 01:38:59.718104 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.718079    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.725919 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.725688    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-mqqdw\""
Oct 31 01:38:59.727533 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.727504    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"]
Oct 31 01:38:59.731034 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.731009    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:38:59.743009 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.742981    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"]
Oct 31 01:38:59.757703 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.757667    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"]
Oct 31 01:38:59.774786 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.774743    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hdz5\" (UniqueName: \"kubernetes.io/projected/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-kube-api-access-7hdz5\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.774984 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.774872    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.774984 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.774899    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.775113 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.774991    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.775113 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.775089    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.775224 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.775130    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.775224 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.775198    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.775322 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.775277    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.775322 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.775310    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.775430 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.775396    5650 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9cd0556f-8541-4c36-80d8-8b18241d3ae8-tls-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:38:59.775430 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.775421    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnb6g\" (UniqueName: \"kubernetes.io/projected/9cd0556f-8541-4c36-80d8-8b18241d3ae8-kube-api-access-wnb6g\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:38:59.887531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.886634    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a125e272-0ca1-47ca-9f22-e46a649b3f10-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:38:59.887531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.886696    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a125e272-0ca1-47ca-9f22-e46a649b3f10-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:38:59.887531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.886755    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.887531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.886782    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.887531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.886821    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.887531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.886853    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a125e272-0ca1-47ca-9f22-e46a649b3f10-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:38:59.887531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.886886    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.887531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.886921    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.887531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.886951    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4rzs\" (UniqueName: \"kubernetes.io/projected/a125e272-0ca1-47ca-9f22-e46a649b3f10-kube-api-access-b4rzs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:38:59.887531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.886988    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a125e272-0ca1-47ca-9f22-e46a649b3f10-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:38:59.887531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.887012    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a125e272-0ca1-47ca-9f22-e46a649b3f10-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:38:59.887531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.887045    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.887531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.887080    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.887531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.887105    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hdz5\" (UniqueName: \"kubernetes.io/projected/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-kube-api-access-7hdz5\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.887531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.887131    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a125e272-0ca1-47ca-9f22-e46a649b3f10-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:38:59.887531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.887159    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a125e272-0ca1-47ca-9f22-e46a649b3f10-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:38:59.887531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.887208    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.887531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.887240    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a125e272-0ca1-47ca-9f22-e46a649b3f10-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:38:59.888575 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.888395    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.889047 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.889022    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.890493 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.889378    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.890637 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.889933    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.890721 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.890136    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.900086 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.900023    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.904687 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.902247    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.904687 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.902493    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.906139 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.906115    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hdz5\" (UniqueName: \"kubernetes.io/projected/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-kube-api-access-7hdz5\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6k8js\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:38:59.988106 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.988053    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a125e272-0ca1-47ca-9f22-e46a649b3f10-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:38:59.988106 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.988114    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4rzs\" (UniqueName: \"kubernetes.io/projected/a125e272-0ca1-47ca-9f22-e46a649b3f10-kube-api-access-b4rzs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:38:59.988372 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.988238    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a125e272-0ca1-47ca-9f22-e46a649b3f10-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:38:59.988372 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.988279    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a125e272-0ca1-47ca-9f22-e46a649b3f10-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:38:59.988372 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.988328    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a125e272-0ca1-47ca-9f22-e46a649b3f10-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:38:59.988372 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.988359    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a125e272-0ca1-47ca-9f22-e46a649b3f10-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:38:59.988593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.988399    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a125e272-0ca1-47ca-9f22-e46a649b3f10-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:38:59.988593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.988485    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a125e272-0ca1-47ca-9f22-e46a649b3f10-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:38:59.988593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.988535    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a125e272-0ca1-47ca-9f22-e46a649b3f10-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:38:59.988779 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.988754    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a125e272-0ca1-47ca-9f22-e46a649b3f10-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:38:59.988818 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.988782    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a125e272-0ca1-47ca-9f22-e46a649b3f10-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:38:59.988897 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.988879    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a125e272-0ca1-47ca-9f22-e46a649b3f10-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:38:59.988950 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.988875    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a125e272-0ca1-47ca-9f22-e46a649b3f10-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:38:59.989167 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.989126    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a125e272-0ca1-47ca-9f22-e46a649b3f10-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:38:59.990996 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.990972    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a125e272-0ca1-47ca-9f22-e46a649b3f10-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:38:59.991083 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:38:59.991059    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a125e272-0ca1-47ca-9f22-e46a649b3f10-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:39:00.001942 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:00.001912    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a125e272-0ca1-47ca-9f22-e46a649b3f10-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:39:00.002085 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:00.001982    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4rzs\" (UniqueName: \"kubernetes.io/projected/a125e272-0ca1-47ca-9f22-e46a649b3f10-kube-api-access-b4rzs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xljzj\" (UID: \"a125e272-0ca1-47ca-9f22-e46a649b3f10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:39:00.032995 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:00.032957    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:39:00.046703 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:00.046678    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:39:00.264009 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:00.263975    5650 generic.go:356] "Generic (PLEG): container finished" podID="9cd0556f-8541-4c36-80d8-8b18241d3ae8" containerID="69d8d0934ae3e8435649402dd59b98caa2b2d5d2981750f26b2261ef0fc79bc1" exitCode=0
Oct 31 01:39:00.264499 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:00.264038    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8bb865d4-fvkqf"
Oct 31 01:39:00.264499 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:00.264069    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8bb865d4-fvkqf" event={"ID":"9cd0556f-8541-4c36-80d8-8b18241d3ae8","Type":"ContainerDied","Data":"69d8d0934ae3e8435649402dd59b98caa2b2d5d2981750f26b2261ef0fc79bc1"}
Oct 31 01:39:00.264499 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:00.264120    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8bb865d4-fvkqf" event={"ID":"9cd0556f-8541-4c36-80d8-8b18241d3ae8","Type":"ContainerDied","Data":"71794dd1d69d9ebdda1949e20a1f239a4006cf36b21b89cf2558bba06213612d"}
Oct 31 01:39:00.264499 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:00.264146    5650 scope.go:122] "RemoveContainer" containerID="69d8d0934ae3e8435649402dd59b98caa2b2d5d2981750f26b2261ef0fc79bc1"
Oct 31 01:39:00.275923 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:00.275899    5650 scope.go:122] "RemoveContainer" containerID="69d8d0934ae3e8435649402dd59b98caa2b2d5d2981750f26b2261ef0fc79bc1"
Oct 31 01:39:00.276347 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:39:00.276315    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69d8d0934ae3e8435649402dd59b98caa2b2d5d2981750f26b2261ef0fc79bc1\": container with ID starting with 69d8d0934ae3e8435649402dd59b98caa2b2d5d2981750f26b2261ef0fc79bc1 not found: ID does not exist" containerID="69d8d0934ae3e8435649402dd59b98caa2b2d5d2981750f26b2261ef0fc79bc1"
Oct 31 01:39:00.276487 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:00.276363    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69d8d0934ae3e8435649402dd59b98caa2b2d5d2981750f26b2261ef0fc79bc1"} err="failed to get container status \"69d8d0934ae3e8435649402dd59b98caa2b2d5d2981750f26b2261ef0fc79bc1\": rpc error: code = NotFound desc = could not find container \"69d8d0934ae3e8435649402dd59b98caa2b2d5d2981750f26b2261ef0fc79bc1\": container with ID starting with 69d8d0934ae3e8435649402dd59b98caa2b2d5d2981750f26b2261ef0fc79bc1 not found: ID does not exist"
Oct 31 01:39:00.300802 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:00.300760    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8bb865d4-fvkqf"]
Oct 31 01:39:00.308368 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:00.308335    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8bb865d4-fvkqf"]
Oct 31 01:39:00.335498 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:00.335466    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"]
Oct 31 01:39:00.336412 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:39:00.336380    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8cc67d3_dcd9_4f63_94b1_b45df6a30e3f.slice/crio-122430080a2395fcf27f39050fc64c6d95ef8a8c929ba804060c62ffb8c23960 WatchSource:0}: Error finding container 122430080a2395fcf27f39050fc64c6d95ef8a8c929ba804060c62ffb8c23960: Status 404 returned error can't find the container with id 122430080a2395fcf27f39050fc64c6d95ef8a8c929ba804060c62ffb8c23960
Oct 31 01:39:00.338756 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:00.338724    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:39:00.338834 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:00.338793    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:39:00.338834 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:00.338829    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:39:00.344085 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:00.344055    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"]
Oct 31 01:39:00.348979 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:39:00.348943    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda125e272_0ca1_47ca_9f22_e46a649b3f10.slice/crio-68524194a0310a6b09122c6a8f41d9a6c32a59c6f501d1a36ea5567d0ac2e6f6 WatchSource:0}: Error finding container 68524194a0310a6b09122c6a8f41d9a6c32a59c6f501d1a36ea5567d0ac2e6f6: Status 404 returned error can't find the container with id 68524194a0310a6b09122c6a8f41d9a6c32a59c6f501d1a36ea5567d0ac2e6f6
Oct 31 01:39:00.351599 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:00.351562    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:39:00.351692 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:00.351657    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:39:00.351753 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:00.351706    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:39:01.271992 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:01.271943    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj" event={"ID":"a125e272-0ca1-47ca-9f22-e46a649b3f10","Type":"ContainerStarted","Data":"bb13ac31f5369043d8e4de931018b7a6f2f0f5b9a3e30eeea8b5974fa94f4ecf"}
Oct 31 01:39:01.272524 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:01.272005    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj" event={"ID":"a125e272-0ca1-47ca-9f22-e46a649b3f10","Type":"ContainerStarted","Data":"68524194a0310a6b09122c6a8f41d9a6c32a59c6f501d1a36ea5567d0ac2e6f6"}
Oct 31 01:39:01.273800 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:01.273769    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js" event={"ID":"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f","Type":"ContainerStarted","Data":"5c7214bc981078f01dbad8bf089d15f5625b711aa105ae8916982c91e212b464"}
Oct 31 01:39:01.273927 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:01.273835    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js" event={"ID":"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f","Type":"ContainerStarted","Data":"122430080a2395fcf27f39050fc64c6d95ef8a8c929ba804060c62ffb8c23960"}
Oct 31 01:39:01.392005 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:01.391939    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js" podStartSLOduration=2.391923908 podStartE2EDuration="2.391923908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:01.390619597 +0000 UTC m=+1097.394333887" watchObservedRunningTime="2026-10-31 01:39:01.391923908 +0000 UTC m=+1097.395638198"
Oct 31 01:39:01.392163 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:01.392067    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj" podStartSLOduration=2.392062111 podStartE2EDuration="2.392062111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:01.320698816 +0000 UTC m=+1097.324413111" watchObservedRunningTime="2026-10-31 01:39:01.392062111 +0000 UTC m=+1097.395776401"
Oct 31 01:39:02.042477 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:02.041951    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:39:02.042477 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:02.042099    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6k8js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 01:39:02.042477 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:02.042147    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js" podUID="b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 01:39:02.052480 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:02.051987    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:39:02.052480 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:02.052120    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xljzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 31 01:39:02.052480 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:02.052165    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj" podUID="a125e272-0ca1-47ca-9f22-e46a649b3f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 31 01:39:02.152785 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:02.152746    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cd0556f-8541-4c36-80d8-8b18241d3ae8" path="/var/lib/kubelet/pods/9cd0556f-8541-4c36-80d8-8b18241d3ae8/volumes"
Oct 31 01:39:03.046472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:03.045919    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6k8js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 01:39:03.046472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:03.045990    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js" podUID="b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 01:39:03.048820 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:03.048544    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xljzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 31 01:39:03.048820 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:03.048602    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj" podUID="a125e272-0ca1-47ca-9f22-e46a649b3f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 31 01:39:04.040471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:04.039912    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6k8js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 01:39:04.040471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:04.039996    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js" podUID="b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 01:39:04.051501 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:04.050544    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xljzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 31 01:39:04.051501 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:04.050659    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj" podUID="a125e272-0ca1-47ca-9f22-e46a649b3f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 31 01:39:05.037472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:05.035083    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6k8js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 01:39:05.037472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:05.035170    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js" podUID="b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 01:39:05.052472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:05.051907    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xljzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 31 01:39:05.052472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:05.051974    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj" podUID="a125e272-0ca1-47ca-9f22-e46a649b3f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 31 01:39:06.036962 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:06.036917    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6k8js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 01:39:06.037275 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:06.037242    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js" podUID="b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 01:39:06.049936 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:06.049889    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xljzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 31 01:39:06.050137 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:06.049957    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj" podUID="a125e272-0ca1-47ca-9f22-e46a649b3f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 31 01:39:07.034124 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:07.034078    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6k8js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 01:39:07.034630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:07.034154    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js" podUID="b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 01:39:07.047970 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:07.047927    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xljzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 31 01:39:07.048118 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:07.047997    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj" podUID="a125e272-0ca1-47ca-9f22-e46a649b3f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 31 01:39:08.034219 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:08.034176    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6k8js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 01:39:08.034629 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:08.034251    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js" podUID="b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 01:39:08.047896 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:08.047850    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xljzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 31 01:39:08.048039 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:08.047921    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj" podUID="a125e272-0ca1-47ca-9f22-e46a649b3f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 31 01:39:09.038062 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:09.038029    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:39:09.038569 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:09.038440    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:39:09.039297 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:09.039277    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:39:09.051791 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:09.051759    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:39:09.052071 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:09.052048    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:39:09.052703 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:09.052687    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj"
Oct 31 01:39:28.552933 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:39:28.552891    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:39:45.359612 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.359525    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2"]
Oct 31 01:39:45.364926 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.364899    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2"
Oct 31 01:39:45.374373 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.374342    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2"]
Oct 31 01:39:45.377950 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.377922    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:39:45.378142 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.378005    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:39:45.388594 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.388563    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8"]
Oct 31 01:39:45.392048 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.392024    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4bf7efc6-serving-cert\""
Oct 31 01:39:45.392176 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.392072    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 01:39:45.392176 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.392109    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qjq8v\""
Oct 31 01:39:45.392247 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.392192    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8"
Oct 31 01:39:45.405356 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.405325    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8"]
Oct 31 01:39:45.407298 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.407275    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4bf7efc6-dockercfg-tkgz4\""
Oct 31 01:39:45.427616 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.427581    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f06d5612-2e35-4dc4-af80-a60915382eea-maas-api-tls\") pod \"maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2\" (UID: \"f06d5612-2e35-4dc4-af80-a60915382eea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2"
Oct 31 01:39:45.427900 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.427872    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f06d5612-2e35-4dc4-af80-a60915382eea-openshift-service-ca\") pod \"maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2\" (UID: \"f06d5612-2e35-4dc4-af80-a60915382eea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2"
Oct 31 01:39:45.428065 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.428037    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7850ced7-5d83-4d43-84e9-c88400db6291-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8\" (UID: \"7850ced7-5d83-4d43-84e9-c88400db6291\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8"
Oct 31 01:39:45.428183 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.428133    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f06d5612-2e35-4dc4-af80-a60915382eea-metrics-certs\") pod \"maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2\" (UID: \"f06d5612-2e35-4dc4-af80-a60915382eea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2"
Oct 31 01:39:45.428227 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.428179    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnqlx\" (UniqueName: \"kubernetes.io/projected/7850ced7-5d83-4d43-84e9-c88400db6291-kube-api-access-mnqlx\") pod \"payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8\" (UID: \"7850ced7-5d83-4d43-84e9-c88400db6291\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8"
Oct 31 01:39:45.428288 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.428271    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtpvd\" (UniqueName: \"kubernetes.io/projected/f06d5612-2e35-4dc4-af80-a60915382eea-kube-api-access-vtpvd\") pod \"maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2\" (UID: \"f06d5612-2e35-4dc4-af80-a60915382eea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2"
Oct 31 01:39:45.459390 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.459351    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m"]
Oct 31 01:39:45.463113 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.463083    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m"
Oct 31 01:39:45.476900 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.476866    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m"]
Oct 31 01:39:45.477761 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.477737    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4bf7efc6\""
Oct 31 01:39:45.529374 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.529333    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f06d5612-2e35-4dc4-af80-a60915382eea-maas-api-tls\") pod \"maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2\" (UID: \"f06d5612-2e35-4dc4-af80-a60915382eea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2"
Oct 31 01:39:45.529586 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.529382    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f06d5612-2e35-4dc4-af80-a60915382eea-openshift-service-ca\") pod \"maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2\" (UID: \"f06d5612-2e35-4dc4-af80-a60915382eea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2"
Oct 31 01:39:45.529586 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.529429    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7850ced7-5d83-4d43-84e9-c88400db6291-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8\" (UID: \"7850ced7-5d83-4d43-84e9-c88400db6291\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8"
Oct 31 01:39:45.529586 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.529516    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26fsd\" (UniqueName: \"kubernetes.io/projected/4aebe63d-3854-4222-ad9c-11a6e5c93219-kube-api-access-26fsd\") pod \"payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m\" (UID: \"4aebe63d-3854-4222-ad9c-11a6e5c93219\") " pod="openshift-ingress/payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m"
Oct 31 01:39:45.529586 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.529551    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f06d5612-2e35-4dc4-af80-a60915382eea-metrics-certs\") pod \"maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2\" (UID: \"f06d5612-2e35-4dc4-af80-a60915382eea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2"
Oct 31 01:39:45.529586 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.529585    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aebe63d-3854-4222-ad9c-11a6e5c93219-plugins-config-volume\") pod \"payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m\" (UID: \"4aebe63d-3854-4222-ad9c-11a6e5c93219\") " pod="openshift-ingress/payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m"
Oct 31 01:39:45.529825 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.529616    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnqlx\" (UniqueName: \"kubernetes.io/projected/7850ced7-5d83-4d43-84e9-c88400db6291-kube-api-access-mnqlx\") pod \"payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8\" (UID: \"7850ced7-5d83-4d43-84e9-c88400db6291\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8"
Oct 31 01:39:45.529928 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.529889    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtpvd\" (UniqueName: \"kubernetes.io/projected/f06d5612-2e35-4dc4-af80-a60915382eea-kube-api-access-vtpvd\") pod \"maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2\" (UID: \"f06d5612-2e35-4dc4-af80-a60915382eea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2"
Oct 31 01:39:45.530296 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.530260    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f06d5612-2e35-4dc4-af80-a60915382eea-openshift-service-ca\") pod \"maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2\" (UID: \"f06d5612-2e35-4dc4-af80-a60915382eea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2"
Oct 31 01:39:45.531207 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.531180    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7850ced7-5d83-4d43-84e9-c88400db6291-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8\" (UID: \"7850ced7-5d83-4d43-84e9-c88400db6291\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8"
Oct 31 01:39:45.532634 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.532608    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f06d5612-2e35-4dc4-af80-a60915382eea-metrics-certs\") pod \"maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2\" (UID: \"f06d5612-2e35-4dc4-af80-a60915382eea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2"
Oct 31 01:39:45.532745 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.532633    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f06d5612-2e35-4dc4-af80-a60915382eea-maas-api-tls\") pod \"maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2\" (UID: \"f06d5612-2e35-4dc4-af80-a60915382eea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2"
Oct 31 01:39:45.543182 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.543150    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnqlx\" (UniqueName: \"kubernetes.io/projected/7850ced7-5d83-4d43-84e9-c88400db6291-kube-api-access-mnqlx\") pod \"payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8\" (UID: \"7850ced7-5d83-4d43-84e9-c88400db6291\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8"
Oct 31 01:39:45.543406 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.543381    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtpvd\" (UniqueName: \"kubernetes.io/projected/f06d5612-2e35-4dc4-af80-a60915382eea-kube-api-access-vtpvd\") pod \"maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2\" (UID: \"f06d5612-2e35-4dc4-af80-a60915382eea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2"
Oct 31 01:39:45.631357 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.631256    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26fsd\" (UniqueName: \"kubernetes.io/projected/4aebe63d-3854-4222-ad9c-11a6e5c93219-kube-api-access-26fsd\") pod \"payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m\" (UID: \"4aebe63d-3854-4222-ad9c-11a6e5c93219\") " pod="openshift-ingress/payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m"
Oct 31 01:39:45.631357 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.631307    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aebe63d-3854-4222-ad9c-11a6e5c93219-plugins-config-volume\") pod \"payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m\" (UID: \"4aebe63d-3854-4222-ad9c-11a6e5c93219\") " pod="openshift-ingress/payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m"
Oct 31 01:39:45.632213 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.632187    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aebe63d-3854-4222-ad9c-11a6e5c93219-plugins-config-volume\") pod \"payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m\" (UID: \"4aebe63d-3854-4222-ad9c-11a6e5c93219\") " pod="openshift-ingress/payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m"
Oct 31 01:39:45.644631 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.644600    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26fsd\" (UniqueName: \"kubernetes.io/projected/4aebe63d-3854-4222-ad9c-11a6e5c93219-kube-api-access-26fsd\") pod \"payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m\" (UID: \"4aebe63d-3854-4222-ad9c-11a6e5c93219\") " pod="openshift-ingress/payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m"
Oct 31 01:39:45.678119 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.678087    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2"
Oct 31 01:39:45.706603 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.706567    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8"
Oct 31 01:39:45.777413 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.777140    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m"
Oct 31 01:39:45.963822 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:45.962709    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2"]
Oct 31 01:39:45.965611 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:39:45.965579    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06d5612_2e35_4dc4_af80_a60915382eea.slice/crio-fa6f738d6c95b712461a364f1f801b80f89d97be4e4973cede5e14039298fa0c WatchSource:0}: Error finding container fa6f738d6c95b712461a364f1f801b80f89d97be4e4973cede5e14039298fa0c: Status 404 returned error can't find the container with id fa6f738d6c95b712461a364f1f801b80f89d97be4e4973cede5e14039298fa0c
Oct 31 01:39:46.000329 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.000295    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8"]
Oct 31 01:39:46.002102 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:39:46.002070    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7850ced7_5d83_4d43_84e9_c88400db6291.slice/crio-b10ac224e81c26bf938fc13b45c672eea849a844c779bf33f9b7a9a1db540799 WatchSource:0}: Error finding container b10ac224e81c26bf938fc13b45c672eea849a844c779bf33f9b7a9a1db540799: Status 404 returned error can't find the container with id b10ac224e81c26bf938fc13b45c672eea849a844c779bf33f9b7a9a1db540799
Oct 31 01:39:46.077525 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.077494    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m"]
Oct 31 01:39:46.079495 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:39:46.079460    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aebe63d_3854_4222_ad9c_11a6e5c93219.slice/crio-3b58fe18b2641cb8ff66788de5b7503fa76e3b1685163e20bf86d9543566a71c WatchSource:0}: Error finding container 3b58fe18b2641cb8ff66788de5b7503fa76e3b1685163e20bf86d9543566a71c: Status 404 returned error can't find the container with id 3b58fe18b2641cb8ff66788de5b7503fa76e3b1685163e20bf86d9543566a71c
Oct 31 01:39:46.183845 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.183808    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n"]
Oct 31 01:39:46.187986 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.187955    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n"
Oct 31 01:39:46.196835 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.196776    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6e7ff10a-dockercfg-22mpm\""
Oct 31 01:39:46.200341 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.199685    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n"]
Oct 31 01:39:46.220293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.220212    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6"]
Oct 31 01:39:46.224034 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.224013    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6"
Oct 31 01:39:46.231100 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.231070    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6"]
Oct 31 01:39:46.238820 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.238793    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9hxc\" (UniqueName: \"kubernetes.io/projected/075829fb-6012-4689-9528-0b97bbb3668b-kube-api-access-j9hxc\") pod \"payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6\" (UID: \"075829fb-6012-4689-9528-0b97bbb3668b\") " pod="openshift-ingress/payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6"
Oct 31 01:39:46.238949 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.238828    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/075829fb-6012-4689-9528-0b97bbb3668b-plugins-config-volume\") pod \"payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6\" (UID: \"075829fb-6012-4689-9528-0b97bbb3668b\") " pod="openshift-ingress/payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6"
Oct 31 01:39:46.238949 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.238853    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0733b3e-2e93-4adb-8f18-1c91f35451c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n\" (UID: \"c0733b3e-2e93-4adb-8f18-1c91f35451c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n"
Oct 31 01:39:46.238949 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.238916    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whh4r\" (UniqueName: \"kubernetes.io/projected/c0733b3e-2e93-4adb-8f18-1c91f35451c8-kube-api-access-whh4r\") pod \"payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n\" (UID: \"c0733b3e-2e93-4adb-8f18-1c91f35451c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n"
Oct 31 01:39:46.245298 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.245276    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6e7ff10a\""
Oct 31 01:39:46.340655 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.340616    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9hxc\" (UniqueName: \"kubernetes.io/projected/075829fb-6012-4689-9528-0b97bbb3668b-kube-api-access-j9hxc\") pod \"payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6\" (UID: \"075829fb-6012-4689-9528-0b97bbb3668b\") " pod="openshift-ingress/payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6"
Oct 31 01:39:46.341326 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.341291    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/075829fb-6012-4689-9528-0b97bbb3668b-plugins-config-volume\") pod \"payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6\" (UID: \"075829fb-6012-4689-9528-0b97bbb3668b\") " pod="openshift-ingress/payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6"
Oct 31 01:39:46.341548 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.341525    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0733b3e-2e93-4adb-8f18-1c91f35451c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n\" (UID: \"c0733b3e-2e93-4adb-8f18-1c91f35451c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n"
Oct 31 01:39:46.341645 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.341564    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whh4r\" (UniqueName: \"kubernetes.io/projected/c0733b3e-2e93-4adb-8f18-1c91f35451c8-kube-api-access-whh4r\") pod \"payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n\" (UID: \"c0733b3e-2e93-4adb-8f18-1c91f35451c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n"
Oct 31 01:39:46.342359 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.342318    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/075829fb-6012-4689-9528-0b97bbb3668b-plugins-config-volume\") pod \"payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6\" (UID: \"075829fb-6012-4689-9528-0b97bbb3668b\") " pod="openshift-ingress/payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6"
Oct 31 01:39:46.342685 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.342660    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0733b3e-2e93-4adb-8f18-1c91f35451c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n\" (UID: \"c0733b3e-2e93-4adb-8f18-1c91f35451c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n"
Oct 31 01:39:46.382682 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.382647    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9hxc\" (UniqueName: \"kubernetes.io/projected/075829fb-6012-4689-9528-0b97bbb3668b-kube-api-access-j9hxc\") pod \"payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6\" (UID: \"075829fb-6012-4689-9528-0b97bbb3668b\") " pod="openshift-ingress/payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6"
Oct 31 01:39:46.385342 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.385320    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whh4r\" (UniqueName: \"kubernetes.io/projected/c0733b3e-2e93-4adb-8f18-1c91f35451c8-kube-api-access-whh4r\") pod \"payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n\" (UID: \"c0733b3e-2e93-4adb-8f18-1c91f35451c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n"
Oct 31 01:39:46.504600 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.504485    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n"
Oct 31 01:39:46.541396 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.541137    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6"
Oct 31 01:39:46.559791 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.559168    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m" event={"ID":"4aebe63d-3854-4222-ad9c-11a6e5c93219","Type":"ContainerStarted","Data":"5b58f100694b550ec026d436a199414c2a02c68f11d4fd309c59080c18440ce9"}
Oct 31 01:39:46.559791 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.559241    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m" event={"ID":"4aebe63d-3854-4222-ad9c-11a6e5c93219","Type":"ContainerStarted","Data":"3b58fe18b2641cb8ff66788de5b7503fa76e3b1685163e20bf86d9543566a71c"}
Oct 31 01:39:46.560546 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.560170    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m"
Oct 31 01:39:46.563009 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.562973    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8" event={"ID":"7850ced7-5d83-4d43-84e9-c88400db6291","Type":"ContainerStarted","Data":"9bdd463555093d509fd8ab9fd69a6e559af372fce566f3d909d0792fc7efc608"}
Oct 31 01:39:46.563122 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.563038    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8" event={"ID":"7850ced7-5d83-4d43-84e9-c88400db6291","Type":"ContainerStarted","Data":"b10ac224e81c26bf938fc13b45c672eea849a844c779bf33f9b7a9a1db540799"}
Oct 31 01:39:46.564162 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.564135    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8"
Oct 31 01:39:46.565983 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.565953    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2" event={"ID":"f06d5612-2e35-4dc4-af80-a60915382eea","Type":"ContainerStarted","Data":"fa6f738d6c95b712461a364f1f801b80f89d97be4e4973cede5e14039298fa0c"}
Oct 31 01:39:46.595494 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.594952    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m" podStartSLOduration=1.594935113 podStartE2EDuration="1.594935113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:46.593473962 +0000 UTC m=+1142.597188254" watchObservedRunningTime="2026-10-31 01:39:46.594935113 +0000 UTC m=+1142.598649403"
Oct 31 01:39:46.720140 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.720077    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8" podStartSLOduration=1.720058783 podStartE2EDuration="1.720058783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:46.626230149 +0000 UTC m=+1142.629944441" watchObservedRunningTime="2026-10-31 01:39:46.720058783 +0000 UTC m=+1142.723773077"
Oct 31 01:39:46.722117 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.722090    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs"]
Oct 31 01:39:46.725918 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.725896    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs"
Oct 31 01:39:46.744506 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.744472    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs"]
Oct 31 01:39:46.744666 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.744529    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-aba34286-dockercfg-dtbpc\""
Oct 31 01:39:46.745384 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.745355    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0616938c-466b-41eb-ac2b-9e66fa2bfa46-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs\" (UID: \"0616938c-466b-41eb-ac2b-9e66fa2bfa46\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs"
Oct 31 01:39:46.745509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.745400    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9f9q\" (UniqueName: \"kubernetes.io/projected/0616938c-466b-41eb-ac2b-9e66fa2bfa46-kube-api-access-n9f9q\") pod \"payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs\" (UID: \"0616938c-466b-41eb-ac2b-9e66fa2bfa46\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs"
Oct 31 01:39:46.748400 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.748372    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-aba34286\""
Oct 31 01:39:46.754354 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.754324    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx"]
Oct 31 01:39:46.758247 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.758180    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx"
Oct 31 01:39:46.775908 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.775876    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx"]
Oct 31 01:39:46.794849 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.794818    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n"]
Oct 31 01:39:46.845191 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.845154    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6"]
Oct 31 01:39:46.846406 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.846181    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0616938c-466b-41eb-ac2b-9e66fa2bfa46-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs\" (UID: \"0616938c-466b-41eb-ac2b-9e66fa2bfa46\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs"
Oct 31 01:39:46.846406 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.846236    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9f9q\" (UniqueName: \"kubernetes.io/projected/0616938c-466b-41eb-ac2b-9e66fa2bfa46-kube-api-access-n9f9q\") pod \"payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs\" (UID: \"0616938c-466b-41eb-ac2b-9e66fa2bfa46\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs"
Oct 31 01:39:46.846406 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.846313    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30c67387-a038-4a9c-a615-7989542d2b6b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx\" (UID: \"30c67387-a038-4a9c-a615-7989542d2b6b\") " pod="openshift-ingress/payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx"
Oct 31 01:39:46.846406 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.846357    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtv9v\" (UniqueName: \"kubernetes.io/projected/30c67387-a038-4a9c-a615-7989542d2b6b-kube-api-access-vtv9v\") pod \"payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx\" (UID: \"30c67387-a038-4a9c-a615-7989542d2b6b\") " pod="openshift-ingress/payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx"
Oct 31 01:39:46.846962 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:39:46.846933    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod075829fb_6012_4689_9528_0b97bbb3668b.slice/crio-54a1e58e07b5ab967844fdbbc4eed47c6ccf6c1336597e72e26b3cb0303fd87d WatchSource:0}: Error finding container 54a1e58e07b5ab967844fdbbc4eed47c6ccf6c1336597e72e26b3cb0303fd87d: Status 404 returned error can't find the container with id 54a1e58e07b5ab967844fdbbc4eed47c6ccf6c1336597e72e26b3cb0303fd87d
Oct 31 01:39:46.847168 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.847143    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0616938c-466b-41eb-ac2b-9e66fa2bfa46-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs\" (UID: \"0616938c-466b-41eb-ac2b-9e66fa2bfa46\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs"
Oct 31 01:39:46.867794 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.867768    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9f9q\" (UniqueName: \"kubernetes.io/projected/0616938c-466b-41eb-ac2b-9e66fa2bfa46-kube-api-access-n9f9q\") pod \"payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs\" (UID: \"0616938c-466b-41eb-ac2b-9e66fa2bfa46\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs"
Oct 31 01:39:46.948027 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.947992    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30c67387-a038-4a9c-a615-7989542d2b6b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx\" (UID: \"30c67387-a038-4a9c-a615-7989542d2b6b\") " pod="openshift-ingress/payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx"
Oct 31 01:39:46.948222 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.948041    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtv9v\" (UniqueName: \"kubernetes.io/projected/30c67387-a038-4a9c-a615-7989542d2b6b-kube-api-access-vtv9v\") pod \"payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx\" (UID: \"30c67387-a038-4a9c-a615-7989542d2b6b\") " pod="openshift-ingress/payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx"
Oct 31 01:39:46.948845 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.948819    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30c67387-a038-4a9c-a615-7989542d2b6b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx\" (UID: \"30c67387-a038-4a9c-a615-7989542d2b6b\") " pod="openshift-ingress/payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx"
Oct 31 01:39:46.961323 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:46.961289    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtv9v\" (UniqueName: \"kubernetes.io/projected/30c67387-a038-4a9c-a615-7989542d2b6b-kube-api-access-vtv9v\") pod \"payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx\" (UID: \"30c67387-a038-4a9c-a615-7989542d2b6b\") " pod="openshift-ingress/payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx"
Oct 31 01:39:47.042172 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:47.042087    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs"
Oct 31 01:39:47.080115 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:47.079856    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx"
Oct 31 01:39:47.334250 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:47.334218    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs"]
Oct 31 01:39:47.336303 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:39:47.336269    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0616938c_466b_41eb_ac2b_9e66fa2bfa46.slice/crio-b353b3c8bfa1ff4055d4d0c66008ebd9e386cb8cfa61be55d843faa3ea327626 WatchSource:0}: Error finding container b353b3c8bfa1ff4055d4d0c66008ebd9e386cb8cfa61be55d843faa3ea327626: Status 404 returned error can't find the container with id b353b3c8bfa1ff4055d4d0c66008ebd9e386cb8cfa61be55d843faa3ea327626
Oct 31 01:39:47.387153 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:47.387079    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx"]
Oct 31 01:39:47.574498 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:47.573544    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx" event={"ID":"30c67387-a038-4a9c-a615-7989542d2b6b","Type":"ContainerStarted","Data":"3a06acb7ba2162c862e058c0619ab983fe647c7c84f6dced515948fc6f6f738d"}
Oct 31 01:39:47.574498 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:47.573598    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx" event={"ID":"30c67387-a038-4a9c-a615-7989542d2b6b","Type":"ContainerStarted","Data":"7cf28822f352aed93c07addcaab9d52f87379a4cce42e01c3e9a52ae2d8c2b7a"}
Oct 31 01:39:47.574498 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:47.574332    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx"
Oct 31 01:39:47.577265 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:47.576479    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6" event={"ID":"075829fb-6012-4689-9528-0b97bbb3668b","Type":"ContainerStarted","Data":"f711cf0356c0b7db0ff428ae55524495b6608789c081723cd394a3f6a28fffee"}
Oct 31 01:39:47.577265 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:47.576537    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6" event={"ID":"075829fb-6012-4689-9528-0b97bbb3668b","Type":"ContainerStarted","Data":"54a1e58e07b5ab967844fdbbc4eed47c6ccf6c1336597e72e26b3cb0303fd87d"}
Oct 31 01:39:47.577265 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:47.577192    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6"
Oct 31 01:39:47.578849 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:47.578816    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs" event={"ID":"0616938c-466b-41eb-ac2b-9e66fa2bfa46","Type":"ContainerStarted","Data":"38f90fcd6896b9443ee29a05b041e0e094616291c79291293e276e21b4eff345"}
Oct 31 01:39:47.578977 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:47.578861    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs" event={"ID":"0616938c-466b-41eb-ac2b-9e66fa2bfa46","Type":"ContainerStarted","Data":"b353b3c8bfa1ff4055d4d0c66008ebd9e386cb8cfa61be55d843faa3ea327626"}
Oct 31 01:39:47.578977 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:47.578892    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs"
Oct 31 01:39:47.580813 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:47.580791    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n" event={"ID":"c0733b3e-2e93-4adb-8f18-1c91f35451c8","Type":"ContainerStarted","Data":"e48d6a033d3940f008e425af7853196dae1578875313e5eca9a7d5190af7719b"}
Oct 31 01:39:47.581055 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:47.581041    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n"
Oct 31 01:39:47.581210 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:47.581197    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n" event={"ID":"c0733b3e-2e93-4adb-8f18-1c91f35451c8","Type":"ContainerStarted","Data":"7450432a2afdf34f6164543bab5355279bc4c566ec9573c68422879d49946f8d"}
Oct 31 01:39:47.598911 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:47.598796    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx" podStartSLOduration=1.5987768 podStartE2EDuration="1.5987768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:47.598127086 +0000 UTC m=+1143.601841372" watchObservedRunningTime="2026-10-31 01:39:47.5987768 +0000 UTC m=+1143.602491092"
Oct 31 01:39:47.622282 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:47.622219    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs" podStartSLOduration=1.6222041969999998 podStartE2EDuration="1.622204197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:47.621780797 +0000 UTC m=+1143.625495090" watchObservedRunningTime="2026-10-31 01:39:47.622204197 +0000 UTC m=+1143.625918489"
Oct 31 01:39:47.680052 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:47.679979    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6" podStartSLOduration=1.679955841 podStartE2EDuration="1.679955841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:47.646500912 +0000 UTC m=+1143.650215205" watchObservedRunningTime="2026-10-31 01:39:47.679955841 +0000 UTC m=+1143.683670132"
Oct 31 01:39:47.682378 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:47.682326    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n" podStartSLOduration=1.682313655 podStartE2EDuration="1.682313655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:47.67846461 +0000 UTC m=+1143.682178902" watchObservedRunningTime="2026-10-31 01:39:47.682313655 +0000 UTC m=+1143.686027945"
Oct 31 01:39:48.046077 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:48.046042    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9"]
Oct 31 01:39:48.049602 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:48.049575    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9"
Oct 31 01:39:48.060129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:48.060098    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6198e156\""
Oct 31 01:39:48.060129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:48.060099    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6198e156-dockercfg-4mnb6\""
Oct 31 01:39:48.060909 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:48.060880    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9"]
Oct 31 01:39:48.065902 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:48.065846    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0274a049-314d-4de4-8424-ca88f05ee373-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9\" (UID: \"0274a049-314d-4de4-8424-ca88f05ee373\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9"
Oct 31 01:39:48.066509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:48.066018    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhnk8\" (UniqueName: \"kubernetes.io/projected/0274a049-314d-4de4-8424-ca88f05ee373-kube-api-access-vhnk8\") pod \"payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9\" (UID: \"0274a049-314d-4de4-8424-ca88f05ee373\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9"
Oct 31 01:39:48.167146 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:48.167112    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0274a049-314d-4de4-8424-ca88f05ee373-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9\" (UID: \"0274a049-314d-4de4-8424-ca88f05ee373\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9"
Oct 31 01:39:48.167349 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:48.167191    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhnk8\" (UniqueName: \"kubernetes.io/projected/0274a049-314d-4de4-8424-ca88f05ee373-kube-api-access-vhnk8\") pod \"payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9\" (UID: \"0274a049-314d-4de4-8424-ca88f05ee373\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9"
Oct 31 01:39:48.167923 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:48.167897    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0274a049-314d-4de4-8424-ca88f05ee373-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9\" (UID: \"0274a049-314d-4de4-8424-ca88f05ee373\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9"
Oct 31 01:39:48.180958 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:48.180931    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhnk8\" (UniqueName: \"kubernetes.io/projected/0274a049-314d-4de4-8424-ca88f05ee373-kube-api-access-vhnk8\") pod \"payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9\" (UID: \"0274a049-314d-4de4-8424-ca88f05ee373\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9"
Oct 31 01:39:48.366881 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:48.366797    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9"
Oct 31 01:39:48.587838 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:48.587781    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2" event={"ID":"f06d5612-2e35-4dc4-af80-a60915382eea","Type":"ContainerStarted","Data":"674f930d497a30393bf8849eb4358fcbbe1b76dd3e50da7b99c4f7e3940c3240"}
Oct 31 01:39:48.588249 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:48.587884    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2"
Oct 31 01:39:48.652922 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:48.652856    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2" podStartSLOduration=1.548219977 podStartE2EDuration="3.652833722s" totalImagesPullingTime="2.104613745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:48.612288195 +0000 UTC m=+1144.616002487" watchObservedRunningTime="2026-10-31 01:39:48.652833722 +0000 UTC m=+1144.656548013"
Oct 31 01:39:48.653205 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:48.653179    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9"]
Oct 31 01:39:48.654580 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:39:48.654554    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0274a049_314d_4de4_8424_ca88f05ee373.slice/crio-98352dec7ded8b0e86e64e3b28752a0e3757a2384fc4ec9bedb497af760076c5 WatchSource:0}: Error finding container 98352dec7ded8b0e86e64e3b28752a0e3757a2384fc4ec9bedb497af760076c5: Status 404 returned error can't find the container with id 98352dec7ded8b0e86e64e3b28752a0e3757a2384fc4ec9bedb497af760076c5
Oct 31 01:39:49.595372 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:49.595308    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9" event={"ID":"0274a049-314d-4de4-8424-ca88f05ee373","Type":"ContainerStarted","Data":"7769f175eaba3dc4062044fd141f74eccedc9a0aa0b8a6cfd6f4b5ebdd286d84"}
Oct 31 01:39:49.595924 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:49.595386    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9" event={"ID":"0274a049-314d-4de4-8424-ca88f05ee373","Type":"ContainerStarted","Data":"98352dec7ded8b0e86e64e3b28752a0e3757a2384fc4ec9bedb497af760076c5"}
Oct 31 01:39:49.618504 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:49.618429    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9" podStartSLOduration=1.618416312 podStartE2EDuration="1.618416312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:49.617117622 +0000 UTC m=+1145.620831918" watchObservedRunningTime="2026-10-31 01:39:49.618416312 +0000 UTC m=+1145.622130603"
Oct 31 01:39:50.601463 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:50.601407    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9"
Oct 31 01:39:51.404157 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.404123    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"]
Oct 31 01:39:51.408321 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.408303    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.413162 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.413140    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-4db52a49-openshift-default-dockercfg-pqwtv\""
Oct 31 01:39:51.423153 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.423120    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"]
Oct 31 01:39:51.503773 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.503725    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-podinfo\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.503773 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.503777    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-credential-socket\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.504092 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.503836    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-workload-socket\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.504092 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.504039    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-workload-certs\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.504211 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.504094    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjt5z\" (UniqueName: \"kubernetes.io/projected/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-kube-api-access-bjt5z\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.504211 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.504192    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-envoy\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.504360 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.504247    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-token\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.504360 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.504297    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-data\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.504533 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.504349    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istiod-ca-cert\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.605221 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.605179    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjt5z\" (UniqueName: \"kubernetes.io/projected/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-kube-api-access-bjt5z\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.605742 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.605235    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-envoy\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.605742 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.605263    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-token\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.605742 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.605296    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-data\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.605742 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.605325    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istiod-ca-cert\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.605742 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.605460    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-podinfo\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.605742 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.605489    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-credential-socket\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.605742 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.605561    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-workload-socket\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.605742 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.605605    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-workload-certs\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.606264 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.606175    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-workload-certs\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.606398 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.606371    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istiod-ca-cert\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.606736 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.606698    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-workload-socket\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.606853 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.606738    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-data\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.606853 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.606833    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-credential-socket\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.608696 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.608666    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-envoy\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.609297 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.609270    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-podinfo\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.619089 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.619058    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-token\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.619319 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.619300    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjt5z\" (UniqueName: \"kubernetes.io/projected/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-kube-api-access-bjt5z\") pod \"e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") " pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:51.730175 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:51.730138    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:52.016968 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:52.016939    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"]
Oct 31 01:39:52.017971 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:39:52.017938    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85900b2_fa9d_42d9_bd0f_70425dbcfda7.slice/crio-0bc920952c426fbccf63d528260ceb5770e1a527c0f8d15d5c7771e917f57d67 WatchSource:0}: Error finding container 0bc920952c426fbccf63d528260ceb5770e1a527c0f8d15d5c7771e917f57d67: Status 404 returned error can't find the container with id 0bc920952c426fbccf63d528260ceb5770e1a527c0f8d15d5c7771e917f57d67
Oct 31 01:39:52.020486 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:52.020429    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:39:52.020571 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:52.020520    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:39:52.020571 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:52.020549    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:39:52.619471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:52.618893    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg" event={"ID":"a85900b2-fa9d-42d9-bd0f-70425dbcfda7","Type":"ContainerStarted","Data":"f5abb7c6c0565747c41cc72a9d0028c055b042db194064ca6b7f614209cb33ca"}
Oct 31 01:39:52.619471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:52.618968    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg" event={"ID":"a85900b2-fa9d-42d9-bd0f-70425dbcfda7","Type":"ContainerStarted","Data":"0bc920952c426fbccf63d528260ceb5770e1a527c0f8d15d5c7771e917f57d67"}
Oct 31 01:39:52.651729 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:52.651671    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg" podStartSLOduration=1.6516550250000002 podStartE2EDuration="1.651655025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:52.6507471 +0000 UTC m=+1148.654461391" watchObservedRunningTime="2026-10-31 01:39:52.651655025 +0000 UTC m=+1148.655369315"
Oct 31 01:39:52.731197 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:52.731143    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:53.735677 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:53.735640    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:54.601395 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:54.601359    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2"
Oct 31 01:39:54.630950 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:54.630918    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:54.631803 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:54.631784    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:39:55.471716 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:55.471682    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq"]
Oct 31 01:39:55.475516 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:55.475496    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq"
Oct 31 01:39:55.486858 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:55.486816    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-4db52a49\""
Oct 31 01:39:55.488897 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:55.488870    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq"]
Oct 31 01:39:55.549015 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:55.548978    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/148e0558-73c9-40e3-a412-594d1b9a6c72-plugins-config-volume\") pod \"payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq\" (UID: \"148e0558-73c9-40e3-a412-594d1b9a6c72\") " pod="openshift-ingress/payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq"
Oct 31 01:39:55.549176 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:55.549095    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgh6h\" (UniqueName: \"kubernetes.io/projected/148e0558-73c9-40e3-a412-594d1b9a6c72-kube-api-access-hgh6h\") pod \"payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq\" (UID: \"148e0558-73c9-40e3-a412-594d1b9a6c72\") " pod="openshift-ingress/payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq"
Oct 31 01:39:55.669816 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:55.669772    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/148e0558-73c9-40e3-a412-594d1b9a6c72-plugins-config-volume\") pod \"payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq\" (UID: \"148e0558-73c9-40e3-a412-594d1b9a6c72\") " pod="openshift-ingress/payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq"
Oct 31 01:39:55.670018 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:55.669879    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgh6h\" (UniqueName: \"kubernetes.io/projected/148e0558-73c9-40e3-a412-594d1b9a6c72-kube-api-access-hgh6h\") pod \"payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq\" (UID: \"148e0558-73c9-40e3-a412-594d1b9a6c72\") " pod="openshift-ingress/payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq"
Oct 31 01:39:55.671285 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:55.671249    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/148e0558-73c9-40e3-a412-594d1b9a6c72-plugins-config-volume\") pod \"payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq\" (UID: \"148e0558-73c9-40e3-a412-594d1b9a6c72\") " pod="openshift-ingress/payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq"
Oct 31 01:39:55.685179 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:55.685146    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgh6h\" (UniqueName: \"kubernetes.io/projected/148e0558-73c9-40e3-a412-594d1b9a6c72-kube-api-access-hgh6h\") pod \"payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq\" (UID: \"148e0558-73c9-40e3-a412-594d1b9a6c72\") " pod="openshift-ingress/payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq"
Oct 31 01:39:55.793020 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:55.792930    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq"
Oct 31 01:39:56.071582 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:56.071556    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq"]
Oct 31 01:39:56.072868 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:39:56.072842    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148e0558_73c9_40e3_a412_594d1b9a6c72.slice/crio-b78e64809c3de37ab281198c211e43bb5554efa7d946a8b60799b8c652f9494e WatchSource:0}: Error finding container b78e64809c3de37ab281198c211e43bb5554efa7d946a8b60799b8c652f9494e: Status 404 returned error can't find the container with id b78e64809c3de37ab281198c211e43bb5554efa7d946a8b60799b8c652f9494e
Oct 31 01:39:56.681059 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:56.681017    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq" event={"ID":"148e0558-73c9-40e3-a412-594d1b9a6c72","Type":"ContainerStarted","Data":"5993c60c21282b2ee7b3a826877ab4209065fe9a81173c6624085a873f1238d6"}
Oct 31 01:39:56.681569 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:56.681088    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq" event={"ID":"148e0558-73c9-40e3-a412-594d1b9a6c72","Type":"ContainerStarted","Data":"b78e64809c3de37ab281198c211e43bb5554efa7d946a8b60799b8c652f9494e"}
Oct 31 01:39:56.681569 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:56.681276    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq"
Oct 31 01:39:56.701911 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:56.701843    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq" podStartSLOduration=1.701822626 podStartE2EDuration="1.701822626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:56.701133179 +0000 UTC m=+1152.704847471" watchObservedRunningTime="2026-10-31 01:39:56.701822626 +0000 UTC m=+1152.705536917"
Oct 31 01:39:58.552844 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:39:58.552738    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:39:58.591524 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:58.591497    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m"
Oct 31 01:39:58.591682 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:58.591560    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs"
Oct 31 01:39:58.591722 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:58.591697    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8"
Oct 31 01:39:59.598517 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:59.598486    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n"
Oct 31 01:39:59.598940 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:59.598563    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6"
Oct 31 01:39:59.598940 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:39:59.598885    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx"
Oct 31 01:40:01.612643 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:01.612613    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9"
Oct 31 01:40:02.234043 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:02.234009    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6198e156-jss66"]
Oct 31 01:40:02.238134 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:02.238108    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6198e156-jss66"
Oct 31 01:40:02.243679 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:02.243644    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:40:02.246316 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:02.246292    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:40:02.249658 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:02.247007    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6198e156-jss66"]
Oct 31 01:40:02.343836 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:02.343794    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5669b53-1a4d-4b27-8088-1f36f1842466-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6198e156-jss66\" (UID: \"b5669b53-1a4d-4b27-8088-1f36f1842466\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6198e156-jss66"
Oct 31 01:40:02.344025 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:02.343945    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5669b53-1a4d-4b27-8088-1f36f1842466-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6198e156-jss66\" (UID: \"b5669b53-1a4d-4b27-8088-1f36f1842466\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6198e156-jss66"
Oct 31 01:40:02.344099 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:02.344033    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqxkn\" (UniqueName: \"kubernetes.io/projected/b5669b53-1a4d-4b27-8088-1f36f1842466-kube-api-access-rqxkn\") pod \"maas-api-revoke-keys-e2e-ait-6198e156-jss66\" (UID: \"b5669b53-1a4d-4b27-8088-1f36f1842466\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6198e156-jss66"
Oct 31 01:40:02.445143 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:02.445101    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5669b53-1a4d-4b27-8088-1f36f1842466-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6198e156-jss66\" (UID: \"b5669b53-1a4d-4b27-8088-1f36f1842466\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6198e156-jss66"
Oct 31 01:40:02.445318 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:02.445154    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqxkn\" (UniqueName: \"kubernetes.io/projected/b5669b53-1a4d-4b27-8088-1f36f1842466-kube-api-access-rqxkn\") pod \"maas-api-revoke-keys-e2e-ait-6198e156-jss66\" (UID: \"b5669b53-1a4d-4b27-8088-1f36f1842466\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6198e156-jss66"
Oct 31 01:40:02.445318 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:02.445210    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5669b53-1a4d-4b27-8088-1f36f1842466-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6198e156-jss66\" (UID: \"b5669b53-1a4d-4b27-8088-1f36f1842466\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6198e156-jss66"
Oct 31 01:40:02.445968 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:02.445947    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5669b53-1a4d-4b27-8088-1f36f1842466-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6198e156-jss66\" (UID: \"b5669b53-1a4d-4b27-8088-1f36f1842466\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6198e156-jss66"
Oct 31 01:40:02.447603 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:02.447578    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5669b53-1a4d-4b27-8088-1f36f1842466-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6198e156-jss66\" (UID: \"b5669b53-1a4d-4b27-8088-1f36f1842466\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6198e156-jss66"
Oct 31 01:40:02.459015 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:02.458984    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqxkn\" (UniqueName: \"kubernetes.io/projected/b5669b53-1a4d-4b27-8088-1f36f1842466-kube-api-access-rqxkn\") pod \"maas-api-revoke-keys-e2e-ait-6198e156-jss66\" (UID: \"b5669b53-1a4d-4b27-8088-1f36f1842466\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6198e156-jss66"
Oct 31 01:40:02.556055 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:02.555942    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6198e156-jss66"
Oct 31 01:40:02.832008 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:02.831972    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6198e156-jss66"]
Oct 31 01:40:02.832911 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:40:02.832877    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5669b53_1a4d_4b27_8088_1f36f1842466.slice/crio-ecbfbe601a7585c00564ba2ad45eba10c85e874468e32820c95e611f4e00b8dc WatchSource:0}: Error finding container ecbfbe601a7585c00564ba2ad45eba10c85e874468e32820c95e611f4e00b8dc: Status 404 returned error can't find the container with id ecbfbe601a7585c00564ba2ad45eba10c85e874468e32820c95e611f4e00b8dc
Oct 31 01:40:03.731865 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:03.731821    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6198e156-jss66" event={"ID":"b5669b53-1a4d-4b27-8088-1f36f1842466","Type":"ContainerStarted","Data":"ecbfbe601a7585c00564ba2ad45eba10c85e874468e32820c95e611f4e00b8dc"}
Oct 31 01:40:07.690491 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:07.690440    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq"
Oct 31 01:40:08.950800 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:08.950706    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm"]
Oct 31 01:40:08.954945 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:08.954919    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm"
Oct 31 01:40:08.963648 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:08.963554    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm"]
Oct 31 01:40:09.041119 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:09.041075    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e15b326-9b9b-477f-b077-04ce95d77635-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm\" (UID: \"6e15b326-9b9b-477f-b077-04ce95d77635\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm"
Oct 31 01:40:09.041285 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:09.041131    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln8rg\" (UniqueName: \"kubernetes.io/projected/6e15b326-9b9b-477f-b077-04ce95d77635-kube-api-access-ln8rg\") pod \"maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm\" (UID: \"6e15b326-9b9b-477f-b077-04ce95d77635\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm"
Oct 31 01:40:09.041285 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:09.041213    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e15b326-9b9b-477f-b077-04ce95d77635-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm\" (UID: \"6e15b326-9b9b-477f-b077-04ce95d77635\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm"
Oct 31 01:40:09.142518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:09.142478    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e15b326-9b9b-477f-b077-04ce95d77635-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm\" (UID: \"6e15b326-9b9b-477f-b077-04ce95d77635\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm"
Oct 31 01:40:09.142662 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:09.142531    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln8rg\" (UniqueName: \"kubernetes.io/projected/6e15b326-9b9b-477f-b077-04ce95d77635-kube-api-access-ln8rg\") pod \"maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm\" (UID: \"6e15b326-9b9b-477f-b077-04ce95d77635\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm"
Oct 31 01:40:09.142662 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:09.142566    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e15b326-9b9b-477f-b077-04ce95d77635-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm\" (UID: \"6e15b326-9b9b-477f-b077-04ce95d77635\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm"
Oct 31 01:40:09.143268 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:09.143240    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e15b326-9b9b-477f-b077-04ce95d77635-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm\" (UID: \"6e15b326-9b9b-477f-b077-04ce95d77635\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm"
Oct 31 01:40:09.145092 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:09.145072    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e15b326-9b9b-477f-b077-04ce95d77635-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm\" (UID: \"6e15b326-9b9b-477f-b077-04ce95d77635\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm"
Oct 31 01:40:09.156165 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:09.156136    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln8rg\" (UniqueName: \"kubernetes.io/projected/6e15b326-9b9b-477f-b077-04ce95d77635-kube-api-access-ln8rg\") pod \"maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm\" (UID: \"6e15b326-9b9b-477f-b077-04ce95d77635\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm"
Oct 31 01:40:09.273024 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:09.272985    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm"
Oct 31 01:40:09.561190 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:09.561153    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm"]
Oct 31 01:40:09.566884 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:40:09.566848    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e15b326_9b9b_477f_b077_04ce95d77635.slice/crio-4723c76de7a9755c74f32a3299997c52bc9c9ef5a544842c4443813b800c3041 WatchSource:0}: Error finding container 4723c76de7a9755c74f32a3299997c52bc9c9ef5a544842c4443813b800c3041: Status 404 returned error can't find the container with id 4723c76de7a9755c74f32a3299997c52bc9c9ef5a544842c4443813b800c3041
Oct 31 01:40:09.767747 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:09.767709    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm" event={"ID":"6e15b326-9b9b-477f-b077-04ce95d77635","Type":"ContainerStarted","Data":"4723c76de7a9755c74f32a3299997c52bc9c9ef5a544842c4443813b800c3041"}
Oct 31 01:40:10.672625 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:10.672580    5650 patch_prober.go:29] interesting pod/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:40:10.673059 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:10.672647    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg" podUID="a85900b2-fa9d-42d9-bd0f-70425dbcfda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:40:11.791331 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:11.791286    5650 generic.go:356] "Generic (PLEG): container finished" podID="a85900b2-fa9d-42d9-bd0f-70425dbcfda7" containerID="f5abb7c6c0565747c41cc72a9d0028c055b042db194064ca6b7f614209cb33ca" exitCode=0
Oct 31 01:40:11.791900 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:11.791362    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg" event={"ID":"a85900b2-fa9d-42d9-bd0f-70425dbcfda7","Type":"ContainerDied","Data":"f5abb7c6c0565747c41cc72a9d0028c055b042db194064ca6b7f614209cb33ca"}
Oct 31 01:40:11.791981 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:11.791904    5650 scope.go:122] "RemoveContainer" containerID="f5abb7c6c0565747c41cc72a9d0028c055b042db194064ca6b7f614209cb33ca"
Oct 31 01:40:11.793753 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:11.793679    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:40:11.793868 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:11.793776    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:40:11.793868 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:11.793823    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:40:12.816473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:12.799466    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg" event={"ID":"a85900b2-fa9d-42d9-bd0f-70425dbcfda7","Type":"ContainerStarted","Data":"0b9a295e0870920001f221d7e103023a85549b407987fd9d385c83cb66eea6c8"}
Oct 31 01:40:13.731363 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:13.731318    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:40:13.731574 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:13.731389    5650 patch_prober.go:29] interesting pod/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 31 01:40:13.731574 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:13.731459    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg" podUID="a85900b2-fa9d-42d9-bd0f-70425dbcfda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 31 01:40:14.730749 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:14.730703    5650 patch_prober.go:29] interesting pod/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 31 01:40:14.731155 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:14.730767    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg" podUID="a85900b2-fa9d-42d9-bd0f-70425dbcfda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 31 01:40:14.840687 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:14.840641    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm" event={"ID":"6e15b326-9b9b-477f-b077-04ce95d77635","Type":"ContainerStarted","Data":"2c3f6a92285a1ae445b79ae77b6bd5d4969306f8f9a0d7bf4c012a1b595d332b"}
Oct 31 01:40:14.864573 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:14.864516    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm" podStartSLOduration=2.281771264 podStartE2EDuration="6.864501127s" totalImagesPullingTime="4.582729863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:14.861960853 +0000 UTC m=+1170.865675146" watchObservedRunningTime="2026-10-31 01:40:14.864501127 +0000 UTC m=+1170.868215768"
Oct 31 01:40:15.680354 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:15.679374    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6"]
Oct 31 01:40:15.690892 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:15.690858    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n"]
Oct 31 01:40:15.731623 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:15.731577    5650 patch_prober.go:29] interesting pod/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 31 01:40:15.732041 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:15.731645    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg" podUID="a85900b2-fa9d-42d9-bd0f-70425dbcfda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 31 01:40:15.848528 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:15.848496    5650 generic.go:356] "Generic (PLEG): container finished" podID="b5669b53-1a4d-4b27-8088-1f36f1842466" containerID="9bdb3c1b71e42203dcc0b74ce2383a07214b9cfd509ed97e01094999dd93dc1d" exitCode=0
Oct 31 01:40:15.848711 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:15.848576    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6198e156-jss66" event={"ID":"b5669b53-1a4d-4b27-8088-1f36f1842466","Type":"ContainerDied","Data":"9bdb3c1b71e42203dcc0b74ce2383a07214b9cfd509ed97e01094999dd93dc1d"}
Oct 31 01:40:15.850175 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:15.850146    5650 generic.go:356] "Generic (PLEG): container finished" podID="6e15b326-9b9b-477f-b077-04ce95d77635" containerID="2c3f6a92285a1ae445b79ae77b6bd5d4969306f8f9a0d7bf4c012a1b595d332b" exitCode=0
Oct 31 01:40:15.850306 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:15.850206    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm" event={"ID":"6e15b326-9b9b-477f-b077-04ce95d77635","Type":"ContainerDied","Data":"2c3f6a92285a1ae445b79ae77b6bd5d4969306f8f9a0d7bf4c012a1b595d332b"}
Oct 31 01:40:16.730663 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:16.730618    5650 patch_prober.go:29] interesting pod/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 31 01:40:16.730861 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:16.730691    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg" podUID="a85900b2-fa9d-42d9-bd0f-70425dbcfda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 31 01:40:17.065051 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.065018    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6198e156-jss66"
Oct 31 01:40:17.097048 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.097014    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm"
Oct 31 01:40:17.163210 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.163173    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5669b53-1a4d-4b27-8088-1f36f1842466-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5669b53-1a4d-4b27-8088-1f36f1842466-maas-api-openshift-ca\") pod \"b5669b53-1a4d-4b27-8088-1f36f1842466\" (UID: \"b5669b53-1a4d-4b27-8088-1f36f1842466\") "
Oct 31 01:40:17.163403 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.163218    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5669b53-1a4d-4b27-8088-1f36f1842466-kube-api-access-rqxkn\" (UniqueName: \"kubernetes.io/projected/b5669b53-1a4d-4b27-8088-1f36f1842466-kube-api-access-rqxkn\") pod \"b5669b53-1a4d-4b27-8088-1f36f1842466\" (UID: \"b5669b53-1a4d-4b27-8088-1f36f1842466\") "
Oct 31 01:40:17.163403 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.163271    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e15b326-9b9b-477f-b077-04ce95d77635-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e15b326-9b9b-477f-b077-04ce95d77635-maas-api-xks-ca\") pod \"6e15b326-9b9b-477f-b077-04ce95d77635\" (UID: \"6e15b326-9b9b-477f-b077-04ce95d77635\") "
Oct 31 01:40:17.163403 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.163340    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5669b53-1a4d-4b27-8088-1f36f1842466-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5669b53-1a4d-4b27-8088-1f36f1842466-maas-api-xks-ca\") pod \"b5669b53-1a4d-4b27-8088-1f36f1842466\" (UID: \"b5669b53-1a4d-4b27-8088-1f36f1842466\") "
Oct 31 01:40:17.163403 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.163376    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e15b326-9b9b-477f-b077-04ce95d77635-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e15b326-9b9b-477f-b077-04ce95d77635-maas-api-openshift-ca\") pod \"6e15b326-9b9b-477f-b077-04ce95d77635\" (UID: \"6e15b326-9b9b-477f-b077-04ce95d77635\") "
Oct 31 01:40:17.163641 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.163547    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e15b326-9b9b-477f-b077-04ce95d77635-kube-api-access-ln8rg\" (UniqueName: \"kubernetes.io/projected/6e15b326-9b9b-477f-b077-04ce95d77635-kube-api-access-ln8rg\") pod \"6e15b326-9b9b-477f-b077-04ce95d77635\" (UID: \"6e15b326-9b9b-477f-b077-04ce95d77635\") "
Oct 31 01:40:17.163701 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.163649    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5669b53-1a4d-4b27-8088-1f36f1842466-maas-api-openshift-ca" pod "b5669b53-1a4d-4b27-8088-1f36f1842466" (UID: "b5669b53-1a4d-4b27-8088-1f36f1842466"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:17.163754 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.163729    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e15b326-9b9b-477f-b077-04ce95d77635-maas-api-openshift-ca" pod "6e15b326-9b9b-477f-b077-04ce95d77635" (UID: "6e15b326-9b9b-477f-b077-04ce95d77635"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:17.163966 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.163946    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e15b326-9b9b-477f-b077-04ce95d77635-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:17.164032 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.163968    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5669b53-1a4d-4b27-8088-1f36f1842466-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:17.165806 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.165779    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e15b326-9b9b-477f-b077-04ce95d77635-maas-api-xks-ca" pod "6e15b326-9b9b-477f-b077-04ce95d77635" (UID: "6e15b326-9b9b-477f-b077-04ce95d77635"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:40:17.165927 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.165796    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5669b53-1a4d-4b27-8088-1f36f1842466-maas-api-xks-ca" pod "b5669b53-1a4d-4b27-8088-1f36f1842466" (UID: "b5669b53-1a4d-4b27-8088-1f36f1842466"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:40:17.165927 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.165819    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e15b326-9b9b-477f-b077-04ce95d77635-kube-api-access-ln8rg" pod "6e15b326-9b9b-477f-b077-04ce95d77635" (UID: "6e15b326-9b9b-477f-b077-04ce95d77635"). InnerVolumeSpecName "kube-api-access-ln8rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:17.166015 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.165926    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5669b53-1a4d-4b27-8088-1f36f1842466-kube-api-access-rqxkn" pod "b5669b53-1a4d-4b27-8088-1f36f1842466" (UID: "b5669b53-1a4d-4b27-8088-1f36f1842466"). InnerVolumeSpecName "kube-api-access-rqxkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:17.265040 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.265005    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqxkn\" (UniqueName: \"kubernetes.io/projected/b5669b53-1a4d-4b27-8088-1f36f1842466-kube-api-access-rqxkn\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:17.265040 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.265037    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e15b326-9b9b-477f-b077-04ce95d77635-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:17.265040 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.265049    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5669b53-1a4d-4b27-8088-1f36f1842466-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:17.265291 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.265058    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln8rg\" (UniqueName: \"kubernetes.io/projected/6e15b326-9b9b-477f-b077-04ce95d77635-kube-api-access-ln8rg\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:17.731097 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.731044    5650 patch_prober.go:29] interesting pod/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 31 01:40:17.731304 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.731112    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg" podUID="a85900b2-fa9d-42d9-bd0f-70425dbcfda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 31 01:40:17.863714 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.863674    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm"
Oct 31 01:40:17.863924 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.863685    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm" event={"ID":"6e15b326-9b9b-477f-b077-04ce95d77635","Type":"ContainerDied","Data":"4723c76de7a9755c74f32a3299997c52bc9c9ef5a544842c4443813b800c3041"}
Oct 31 01:40:17.863924 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.863822    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4723c76de7a9755c74f32a3299997c52bc9c9ef5a544842c4443813b800c3041"
Oct 31 01:40:17.865996 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.865965    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6198e156-jss66" event={"ID":"b5669b53-1a4d-4b27-8088-1f36f1842466","Type":"ContainerDied","Data":"ecbfbe601a7585c00564ba2ad45eba10c85e874468e32820c95e611f4e00b8dc"}
Oct 31 01:40:17.866168 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.866024    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecbfbe601a7585c00564ba2ad45eba10c85e874468e32820c95e611f4e00b8dc"
Oct 31 01:40:17.866168 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:17.865967    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6198e156-jss66"
Oct 31 01:40:18.741853 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:18.741821    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:40:18.742253 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:18.742091    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:40:18.742967 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:18.742947    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:40:19.130498 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:19.130402    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq"]
Oct 31 01:40:20.276332 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:20.276298    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m"]
Oct 31 01:40:20.301117 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:20.301075    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8"]
Oct 31 01:40:20.541836 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:20.541747    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx"]
Oct 31 01:40:20.557262 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:20.557230    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs"]
Oct 31 01:40:20.680403 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:20.680322    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6" podUID="075829fb-6012-4689-9528-0b97bbb3668b" containerName="payload-processing" containerID="cri-o://f711cf0356c0b7db0ff428ae55524495b6608789c081723cd394a3f6a28fffee" gracePeriod=25
Oct 31 01:40:20.692098 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:20.692054    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n" podUID="c0733b3e-2e93-4adb-8f18-1c91f35451c8" containerName="payload-pre-processing" containerID="cri-o://e48d6a033d3940f008e425af7853196dae1578875313e5eca9a7d5190af7719b" gracePeriod=25
Oct 31 01:40:20.890313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:20.890128    5650 generic.go:356] "Generic (PLEG): container finished" podID="075829fb-6012-4689-9528-0b97bbb3668b" containerID="f711cf0356c0b7db0ff428ae55524495b6608789c081723cd394a3f6a28fffee" exitCode=0
Oct 31 01:40:20.890313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:20.890237    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6" event={"ID":"075829fb-6012-4689-9528-0b97bbb3668b","Type":"ContainerDied","Data":"f711cf0356c0b7db0ff428ae55524495b6608789c081723cd394a3f6a28fffee"}
Oct 31 01:40:20.892054 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:20.891998    5650 generic.go:356] "Generic (PLEG): container finished" podID="c0733b3e-2e93-4adb-8f18-1c91f35451c8" containerID="e48d6a033d3940f008e425af7853196dae1578875313e5eca9a7d5190af7719b" exitCode=0
Oct 31 01:40:20.892054 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:20.892042    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n" event={"ID":"c0733b3e-2e93-4adb-8f18-1c91f35451c8","Type":"ContainerDied","Data":"e48d6a033d3940f008e425af7853196dae1578875313e5eca9a7d5190af7719b"}
Oct 31 01:40:20.972690 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:20.972659    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6"
Oct 31 01:40:20.994716 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:20.994686    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n"
Oct 31 01:40:21.004963 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.004930    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/075829fb-6012-4689-9528-0b97bbb3668b-kube-api-access-j9hxc\" (UniqueName: \"kubernetes.io/projected/075829fb-6012-4689-9528-0b97bbb3668b-kube-api-access-j9hxc\") pod \"075829fb-6012-4689-9528-0b97bbb3668b\" (UID: \"075829fb-6012-4689-9528-0b97bbb3668b\") "
Oct 31 01:40:21.005164 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.005034    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/075829fb-6012-4689-9528-0b97bbb3668b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/075829fb-6012-4689-9528-0b97bbb3668b-plugins-config-volume\") pod \"075829fb-6012-4689-9528-0b97bbb3668b\" (UID: \"075829fb-6012-4689-9528-0b97bbb3668b\") "
Oct 31 01:40:21.005436 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.005401    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/075829fb-6012-4689-9528-0b97bbb3668b-plugins-config-volume" pod "075829fb-6012-4689-9528-0b97bbb3668b" (UID: "075829fb-6012-4689-9528-0b97bbb3668b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:21.007289 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.007264    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/075829fb-6012-4689-9528-0b97bbb3668b-kube-api-access-j9hxc" pod "075829fb-6012-4689-9528-0b97bbb3668b" (UID: "075829fb-6012-4689-9528-0b97bbb3668b"). InnerVolumeSpecName "kube-api-access-j9hxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:21.106593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.106489    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0733b3e-2e93-4adb-8f18-1c91f35451c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0733b3e-2e93-4adb-8f18-1c91f35451c8-plugins-config-volume\") pod \"c0733b3e-2e93-4adb-8f18-1c91f35451c8\" (UID: \"c0733b3e-2e93-4adb-8f18-1c91f35451c8\") "
Oct 31 01:40:21.106772 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.106631    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0733b3e-2e93-4adb-8f18-1c91f35451c8-kube-api-access-whh4r\" (UniqueName: \"kubernetes.io/projected/c0733b3e-2e93-4adb-8f18-1c91f35451c8-kube-api-access-whh4r\") pod \"c0733b3e-2e93-4adb-8f18-1c91f35451c8\" (UID: \"c0733b3e-2e93-4adb-8f18-1c91f35451c8\") "
Oct 31 01:40:21.106843 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.106814    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9hxc\" (UniqueName: \"kubernetes.io/projected/075829fb-6012-4689-9528-0b97bbb3668b-kube-api-access-j9hxc\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:21.106843 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.106826    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/075829fb-6012-4689-9528-0b97bbb3668b-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:21.106998 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.106972    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0733b3e-2e93-4adb-8f18-1c91f35451c8-plugins-config-volume" pod "c0733b3e-2e93-4adb-8f18-1c91f35451c8" (UID: "c0733b3e-2e93-4adb-8f18-1c91f35451c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:21.108691 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.108667    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0733b3e-2e93-4adb-8f18-1c91f35451c8-kube-api-access-whh4r" pod "c0733b3e-2e93-4adb-8f18-1c91f35451c8" (UID: "c0733b3e-2e93-4adb-8f18-1c91f35451c8"). InnerVolumeSpecName "kube-api-access-whh4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:21.207698 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.207665    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0733b3e-2e93-4adb-8f18-1c91f35451c8-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:21.207698 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.207698    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whh4r\" (UniqueName: \"kubernetes.io/projected/c0733b3e-2e93-4adb-8f18-1c91f35451c8-kube-api-access-whh4r\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:21.221795 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.221758    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2"]
Oct 31 01:40:21.222016 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.221970    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2" podUID="f06d5612-2e35-4dc4-af80-a60915382eea" containerName="maas-api" containerID="cri-o://674f930d497a30393bf8849eb4358fcbbe1b76dd3e50da7b99c4f7e3940c3240" gracePeriod=30
Oct 31 01:40:21.464153 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.464125    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2"
Oct 31 01:40:21.510472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.509795    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f06d5612-2e35-4dc4-af80-a60915382eea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f06d5612-2e35-4dc4-af80-a60915382eea-openshift-service-ca\") pod \"f06d5612-2e35-4dc4-af80-a60915382eea\" (UID: \"f06d5612-2e35-4dc4-af80-a60915382eea\") "
Oct 31 01:40:21.510472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.509895    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f06d5612-2e35-4dc4-af80-a60915382eea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f06d5612-2e35-4dc4-af80-a60915382eea-metrics-certs\") pod \"f06d5612-2e35-4dc4-af80-a60915382eea\" (UID: \"f06d5612-2e35-4dc4-af80-a60915382eea\") "
Oct 31 01:40:21.510472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.510033    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f06d5612-2e35-4dc4-af80-a60915382eea-kube-api-access-vtpvd\" (UniqueName: \"kubernetes.io/projected/f06d5612-2e35-4dc4-af80-a60915382eea-kube-api-access-vtpvd\") pod \"f06d5612-2e35-4dc4-af80-a60915382eea\" (UID: \"f06d5612-2e35-4dc4-af80-a60915382eea\") "
Oct 31 01:40:21.510472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.510073    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f06d5612-2e35-4dc4-af80-a60915382eea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f06d5612-2e35-4dc4-af80-a60915382eea-maas-api-tls\") pod \"f06d5612-2e35-4dc4-af80-a60915382eea\" (UID: \"f06d5612-2e35-4dc4-af80-a60915382eea\") "
Oct 31 01:40:21.517422 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.511621    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f06d5612-2e35-4dc4-af80-a60915382eea-openshift-service-ca" pod "f06d5612-2e35-4dc4-af80-a60915382eea" (UID: "f06d5612-2e35-4dc4-af80-a60915382eea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:21.518854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.518495    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f06d5612-2e35-4dc4-af80-a60915382eea-metrics-certs" pod "f06d5612-2e35-4dc4-af80-a60915382eea" (UID: "f06d5612-2e35-4dc4-af80-a60915382eea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:40:21.518854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.518608    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f06d5612-2e35-4dc4-af80-a60915382eea-maas-api-tls" pod "f06d5612-2e35-4dc4-af80-a60915382eea" (UID: "f06d5612-2e35-4dc4-af80-a60915382eea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:40:21.519996 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.519961    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06d5612-2e35-4dc4-af80-a60915382eea-kube-api-access-vtpvd" pod "f06d5612-2e35-4dc4-af80-a60915382eea" (UID: "f06d5612-2e35-4dc4-af80-a60915382eea"). InnerVolumeSpecName "kube-api-access-vtpvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:21.610933 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.610898    5650 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f06d5612-2e35-4dc4-af80-a60915382eea-metrics-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:21.610933 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.610929    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtpvd\" (UniqueName: \"kubernetes.io/projected/f06d5612-2e35-4dc4-af80-a60915382eea-kube-api-access-vtpvd\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:21.611180 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.610943    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f06d5612-2e35-4dc4-af80-a60915382eea-maas-api-tls\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:21.611180 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.610957    5650 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f06d5612-2e35-4dc4-af80-a60915382eea-openshift-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:21.899549 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.899515    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6"
Oct 31 01:40:21.899731 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.899561    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6" event={"ID":"075829fb-6012-4689-9528-0b97bbb3668b","Type":"ContainerDied","Data":"54a1e58e07b5ab967844fdbbc4eed47c6ccf6c1336597e72e26b3cb0303fd87d"}
Oct 31 01:40:21.899731 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.899629    5650 scope.go:122] "RemoveContainer" containerID="f711cf0356c0b7db0ff428ae55524495b6608789c081723cd394a3f6a28fffee"
Oct 31 01:40:21.900983 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.900945    5650 generic.go:356] "Generic (PLEG): container finished" podID="f06d5612-2e35-4dc4-af80-a60915382eea" containerID="674f930d497a30393bf8849eb4358fcbbe1b76dd3e50da7b99c4f7e3940c3240" exitCode=0
Oct 31 01:40:21.901120 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.900980    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2" event={"ID":"f06d5612-2e35-4dc4-af80-a60915382eea","Type":"ContainerDied","Data":"674f930d497a30393bf8849eb4358fcbbe1b76dd3e50da7b99c4f7e3940c3240"}
Oct 31 01:40:21.901120 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.900997    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2"
Oct 31 01:40:21.901120 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.901027    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2" event={"ID":"f06d5612-2e35-4dc4-af80-a60915382eea","Type":"ContainerDied","Data":"fa6f738d6c95b712461a364f1f801b80f89d97be4e4973cede5e14039298fa0c"}
Oct 31 01:40:21.903364 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.903341    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n" event={"ID":"c0733b3e-2e93-4adb-8f18-1c91f35451c8","Type":"ContainerDied","Data":"7450432a2afdf34f6164543bab5355279bc4c566ec9573c68422879d49946f8d"}
Oct 31 01:40:21.903364 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.903356    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n"
Oct 31 01:40:21.911118 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.911099    5650 scope.go:122] "RemoveContainer" containerID="674f930d497a30393bf8849eb4358fcbbe1b76dd3e50da7b99c4f7e3940c3240"
Oct 31 01:40:21.922268 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.922250    5650 scope.go:122] "RemoveContainer" containerID="674f930d497a30393bf8849eb4358fcbbe1b76dd3e50da7b99c4f7e3940c3240"
Oct 31 01:40:21.922613 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:40:21.922590    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"674f930d497a30393bf8849eb4358fcbbe1b76dd3e50da7b99c4f7e3940c3240\": container with ID starting with 674f930d497a30393bf8849eb4358fcbbe1b76dd3e50da7b99c4f7e3940c3240 not found: ID does not exist" containerID="674f930d497a30393bf8849eb4358fcbbe1b76dd3e50da7b99c4f7e3940c3240"
Oct 31 01:40:21.922716 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.922622    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674f930d497a30393bf8849eb4358fcbbe1b76dd3e50da7b99c4f7e3940c3240"} err="failed to get container status \"674f930d497a30393bf8849eb4358fcbbe1b76dd3e50da7b99c4f7e3940c3240\": rpc error: code = NotFound desc = could not find container \"674f930d497a30393bf8849eb4358fcbbe1b76dd3e50da7b99c4f7e3940c3240\": container with ID starting with 674f930d497a30393bf8849eb4358fcbbe1b76dd3e50da7b99c4f7e3940c3240 not found: ID does not exist"
Oct 31 01:40:21.922716 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.922641    5650 scope.go:122] "RemoveContainer" containerID="e48d6a033d3940f008e425af7853196dae1578875313e5eca9a7d5190af7719b"
Oct 31 01:40:21.930924 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.930886    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6"]
Oct 31 01:40:21.937677 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.937648    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e7ff10a-67cc6fb687-lw4q6"]
Oct 31 01:40:21.955692 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.953334    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n"]
Oct 31 01:40:21.959746 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.959715    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e7ff10a-c6d4f5dff-nz62n"]
Oct 31 01:40:21.973586 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.973551    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2"]
Oct 31 01:40:21.978319 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:21.978288    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4bf7efc6-86676ddbcd-5npk2"]
Oct 31 01:40:22.117362 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.117326    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="075829fb-6012-4689-9528-0b97bbb3668b" path="/var/lib/kubelet/pods/075829fb-6012-4689-9528-0b97bbb3668b/volumes"
Oct 31 01:40:22.117785 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.117770    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0733b3e-2e93-4adb-8f18-1c91f35451c8" path="/var/lib/kubelet/pods/c0733b3e-2e93-4adb-8f18-1c91f35451c8/volumes"
Oct 31 01:40:22.118136 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.118124    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f06d5612-2e35-4dc4-af80-a60915382eea" path="/var/lib/kubelet/pods/f06d5612-2e35-4dc4-af80-a60915382eea/volumes"
Oct 31 01:40:22.438952 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.438861    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9"]
Oct 31 01:40:22.568181 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.568144    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm"]
Oct 31 01:40:22.570734 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.570705    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4db52a49-6qtmm"]
Oct 31 01:40:22.738843 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.738809    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"]
Oct 31 01:40:22.739148 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.739118    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg" podUID="a85900b2-fa9d-42d9-bd0f-70425dbcfda7" containerName="istio-proxy" containerID="cri-o://0b9a295e0870920001f221d7e103023a85549b407987fd9d385c83cb66eea6c8" gracePeriod=30
Oct 31 01:40:22.848168 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.848121    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"]
Oct 31 01:40:22.848931 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.848909    5650 cpu_manager.go:414] "containerMap: removing container" podUID="6e15b326-9b9b-477f-b077-04ce95d77635" containerName="revoke-keys"
Oct 31 01:40:22.849002 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.848935    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e15b326-9b9b-477f-b077-04ce95d77635" containerName="revoke-keys"
Oct 31 01:40:22.849002 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.848967    5650 cpu_manager.go:414] "containerMap: removing container" podUID="075829fb-6012-4689-9528-0b97bbb3668b" containerName="payload-processing"
Oct 31 01:40:22.849002 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.848976    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="075829fb-6012-4689-9528-0b97bbb3668b" containerName="payload-processing"
Oct 31 01:40:22.849002 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.848988    5650 cpu_manager.go:414] "containerMap: removing container" podUID="b5669b53-1a4d-4b27-8088-1f36f1842466" containerName="revoke-keys"
Oct 31 01:40:22.849002 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.848997    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5669b53-1a4d-4b27-8088-1f36f1842466" containerName="revoke-keys"
Oct 31 01:40:22.849160 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.849012    5650 cpu_manager.go:414] "containerMap: removing container" podUID="c0733b3e-2e93-4adb-8f18-1c91f35451c8" containerName="payload-pre-processing"
Oct 31 01:40:22.849160 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.849020    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0733b3e-2e93-4adb-8f18-1c91f35451c8" containerName="payload-pre-processing"
Oct 31 01:40:22.849160 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.849045    5650 cpu_manager.go:414] "containerMap: removing container" podUID="f06d5612-2e35-4dc4-af80-a60915382eea" containerName="maas-api"
Oct 31 01:40:22.849160 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.849053    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06d5612-2e35-4dc4-af80-a60915382eea" containerName="maas-api"
Oct 31 01:40:22.849277 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.849187    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="075829fb-6012-4689-9528-0b97bbb3668b" containerName="payload-processing"
Oct 31 01:40:22.849277 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.849200    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5669b53-1a4d-4b27-8088-1f36f1842466" containerName="revoke-keys"
Oct 31 01:40:22.849277 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.849212    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0733b3e-2e93-4adb-8f18-1c91f35451c8" containerName="payload-pre-processing"
Oct 31 01:40:22.849277 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.849225    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06d5612-2e35-4dc4-af80-a60915382eea" containerName="maas-api"
Oct 31 01:40:22.849277 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.849239    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e15b326-9b9b-477f-b077-04ce95d77635" containerName="revoke-keys"
Oct 31 01:40:22.854373 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.854349    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:22.860120 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.860095    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-6ae519-openshift-default-dockercfg-kmlj9\""
Oct 31 01:40:22.863889 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.863858    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"]
Oct 31 01:40:22.934836 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.934794    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:22.934836 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.934838    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:22.935109 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.934975    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzzb4\" (UniqueName: \"kubernetes.io/projected/cd6018a4-4e7e-410a-892f-6b3f800addd7-kube-api-access-lzzb4\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:22.935109 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.935012    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:22.935213 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.935102    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:22.935213 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.935157    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:22.935310 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.935234    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:22.935361 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.935344    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd6018a4-4e7e-410a-892f-6b3f800addd7-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:22.935394 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:22.935382    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:23.036779 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.036681    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:23.036779 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.036730    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:23.037179 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.037154    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:23.037179 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.037170    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:23.037370 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.036891    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:23.037427 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.037396    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzzb4\" (UniqueName: \"kubernetes.io/projected/cd6018a4-4e7e-410a-892f-6b3f800addd7-kube-api-access-lzzb4\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:23.037491 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.037427    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:23.037551 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.037516    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:23.037602 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.037559    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:23.037657 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.037601    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:23.037712 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.037690    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd6018a4-4e7e-410a-892f-6b3f800addd7-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:23.038272 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.038240    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:23.038386 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.038310    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd6018a4-4e7e-410a-892f-6b3f800addd7-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:23.038572 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.038541    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:23.040498 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.040474    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:23.040749 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.040729    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:23.051267 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.051236    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:23.051545 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.051518    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzzb4\" (UniqueName: \"kubernetes.io/projected/cd6018a4-4e7e-410a-892f-6b3f800addd7-kube-api-access-lzzb4\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:23.169688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.169650    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:23.456590 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.456562    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"]
Oct 31 01:40:23.458340 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:40:23.458307    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd6018a4_4e7e_410a_892f_6b3f800addd7.slice/crio-3a5cb11b4bb9d3888d545e219c2bba2e101d019dd1ba21f017792e0f881fcaeb WatchSource:0}: Error finding container 3a5cb11b4bb9d3888d545e219c2bba2e101d019dd1ba21f017792e0f881fcaeb: Status 404 returned error can't find the container with id 3a5cb11b4bb9d3888d545e219c2bba2e101d019dd1ba21f017792e0f881fcaeb
Oct 31 01:40:23.460607 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.460577    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:40:23.460693 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.460656    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:40:23.460693 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.460687    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:40:23.923523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.923391    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj" event={"ID":"cd6018a4-4e7e-410a-892f-6b3f800addd7","Type":"ContainerStarted","Data":"ef3c2fc9fedc601a7e8676e784a2668eca04022ab412a13713042dde38b7b5e7"}
Oct 31 01:40:23.923523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.923497    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj" event={"ID":"cd6018a4-4e7e-410a-892f-6b3f800addd7","Type":"ContainerStarted","Data":"3a5cb11b4bb9d3888d545e219c2bba2e101d019dd1ba21f017792e0f881fcaeb"}
Oct 31 01:40:23.951194 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:23.951141    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj" podStartSLOduration=1.951126265 podStartE2EDuration="1.951126265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:23.94927603 +0000 UTC m=+1179.952990341" watchObservedRunningTime="2026-10-31 01:40:23.951126265 +0000 UTC m=+1179.954840556"
Oct 31 01:40:24.117887 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.117850    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e15b326-9b9b-477f-b077-04ce95d77635" path="/var/lib/kubelet/pods/6e15b326-9b9b-477f-b077-04ce95d77635/volumes"
Oct 31 01:40:24.131240 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.131192    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq" podUID="148e0558-73c9-40e3-a412-594d1b9a6c72" containerName="payload-processing" containerID="cri-o://5993c60c21282b2ee7b3a826877ab4209065fe9a81173c6624085a873f1238d6" gracePeriod=25
Oct 31 01:40:24.156840 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.156805    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6198e156-jss66"]
Oct 31 01:40:24.161556 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.161516    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6198e156-jss66"]
Oct 31 01:40:24.170121 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.170086    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:24.175676 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.175593    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:24.355732 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.355675    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"]
Oct 31 01:40:24.360204 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.360116    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.373163 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.372679    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-7e5955-openshift-default-dockercfg-t99cw\""
Oct 31 01:40:24.376938 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.376906    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"]
Oct 31 01:40:24.405557 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.405526    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq"
Oct 31 01:40:24.453666 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.453628    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/148e0558-73c9-40e3-a412-594d1b9a6c72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/148e0558-73c9-40e3-a412-594d1b9a6c72-plugins-config-volume\") pod \"148e0558-73c9-40e3-a412-594d1b9a6c72\" (UID: \"148e0558-73c9-40e3-a412-594d1b9a6c72\") "
Oct 31 01:40:24.453845 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.453805    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/148e0558-73c9-40e3-a412-594d1b9a6c72-kube-api-access-hgh6h\" (UniqueName: \"kubernetes.io/projected/148e0558-73c9-40e3-a412-594d1b9a6c72-kube-api-access-hgh6h\") pod \"148e0558-73c9-40e3-a412-594d1b9a6c72\" (UID: \"148e0558-73c9-40e3-a412-594d1b9a6c72\") "
Oct 31 01:40:24.454095 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.454073    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.454155 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.454110    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/148e0558-73c9-40e3-a412-594d1b9a6c72-plugins-config-volume" pod "148e0558-73c9-40e3-a412-594d1b9a6c72" (UID: "148e0558-73c9-40e3-a412-594d1b9a6c72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:24.454223 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.454205    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.454294 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.454239    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.454411 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.454394    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.454474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.454429    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w979g\" (UniqueName: \"kubernetes.io/projected/af0ee471-3ebf-426f-88e8-689ddc123d8e-kube-api-access-w979g\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.454530 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.454474    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.454617 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.454594    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.454719 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.454696    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.454783 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.454735    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af0ee471-3ebf-426f-88e8-689ddc123d8e-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.454917 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.454897    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/148e0558-73c9-40e3-a412-594d1b9a6c72-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:24.456102 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.456075    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/148e0558-73c9-40e3-a412-594d1b9a6c72-kube-api-access-hgh6h" pod "148e0558-73c9-40e3-a412-594d1b9a6c72" (UID: "148e0558-73c9-40e3-a412-594d1b9a6c72"). InnerVolumeSpecName "kube-api-access-hgh6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:24.556800 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.556676    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.556800 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.556735    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af0ee471-3ebf-426f-88e8-689ddc123d8e-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.557054 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.556842    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.557054 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.556931    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.557054 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.556963    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.557054 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.557034    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.557274 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.557073    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w979g\" (UniqueName: \"kubernetes.io/projected/af0ee471-3ebf-426f-88e8-689ddc123d8e-kube-api-access-w979g\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.557274 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.557121    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.557274 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.557161    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.558867 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.557745    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af0ee471-3ebf-426f-88e8-689ddc123d8e-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.558867 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.558030    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.559422 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.559394    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.568188 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.559858    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgh6h\" (UniqueName: \"kubernetes.io/projected/148e0558-73c9-40e3-a412-594d1b9a6c72-kube-api-access-hgh6h\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:24.568188 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.560066    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.568188 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.567774    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.568416 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.568148    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.571597 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.570092    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.581489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.578475    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w979g\" (UniqueName: \"kubernetes.io/projected/af0ee471-3ebf-426f-88e8-689ddc123d8e-kube-api-access-w979g\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.581489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.578946    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.660369 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.660330    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw"]
Oct 31 01:40:24.660850 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.660816    5650 cpu_manager.go:414] "containerMap: removing container" podUID="148e0558-73c9-40e3-a412-594d1b9a6c72" containerName="payload-processing"
Oct 31 01:40:24.660850 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.660831    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="148e0558-73c9-40e3-a412-594d1b9a6c72" containerName="payload-processing"
Oct 31 01:40:24.660962 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.660934    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="148e0558-73c9-40e3-a412-594d1b9a6c72" containerName="payload-processing"
Oct 31 01:40:24.665844 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.665824    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw"
Oct 31 01:40:24.672385 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.671967    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-be6935\""
Oct 31 01:40:24.672589 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.672564    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-be6935-dockercfg-xl9th\""
Oct 31 01:40:24.675553 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.675529    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw"]
Oct 31 01:40:24.683768 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.683740    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:24.708207 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.708129    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p"]
Oct 31 01:40:24.712440 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.712413    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p"
Oct 31 01:40:24.722877 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.722834    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p"]
Oct 31 01:40:24.775520 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.772115    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35f0ddb1-3758-4a14-a2e2-2c7a928b3564-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw\" (UID: \"35f0ddb1-3758-4a14-a2e2-2c7a928b3564\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw"
Oct 31 01:40:24.775520 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.772194    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p\" (UID: \"7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p"
Oct 31 01:40:24.775520 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.772286    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pkl4\" (UniqueName: \"kubernetes.io/projected/35f0ddb1-3758-4a14-a2e2-2c7a928b3564-kube-api-access-6pkl4\") pod \"payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw\" (UID: \"35f0ddb1-3758-4a14-a2e2-2c7a928b3564\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw"
Oct 31 01:40:24.775520 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.772355    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcfcz\" (UniqueName: \"kubernetes.io/projected/7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e-kube-api-access-xcfcz\") pod \"payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p\" (UID: \"7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p"
Oct 31 01:40:24.873240 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.873154    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35f0ddb1-3758-4a14-a2e2-2c7a928b3564-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw\" (UID: \"35f0ddb1-3758-4a14-a2e2-2c7a928b3564\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw"
Oct 31 01:40:24.873240 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.873217    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p\" (UID: \"7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p"
Oct 31 01:40:24.873480 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.873271    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pkl4\" (UniqueName: \"kubernetes.io/projected/35f0ddb1-3758-4a14-a2e2-2c7a928b3564-kube-api-access-6pkl4\") pod \"payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw\" (UID: \"35f0ddb1-3758-4a14-a2e2-2c7a928b3564\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw"
Oct 31 01:40:24.873480 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.873334    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcfcz\" (UniqueName: \"kubernetes.io/projected/7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e-kube-api-access-xcfcz\") pod \"payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p\" (UID: \"7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p"
Oct 31 01:40:24.873937 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.873916    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35f0ddb1-3758-4a14-a2e2-2c7a928b3564-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw\" (UID: \"35f0ddb1-3758-4a14-a2e2-2c7a928b3564\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw"
Oct 31 01:40:24.874104 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.874086    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p\" (UID: \"7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p"
Oct 31 01:40:24.886827 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.886788    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcfcz\" (UniqueName: \"kubernetes.io/projected/7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e-kube-api-access-xcfcz\") pod \"payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p\" (UID: \"7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p"
Oct 31 01:40:24.886827 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.886807    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pkl4\" (UniqueName: \"kubernetes.io/projected/35f0ddb1-3758-4a14-a2e2-2c7a928b3564-kube-api-access-6pkl4\") pod \"payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw\" (UID: \"35f0ddb1-3758-4a14-a2e2-2c7a928b3564\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw"
Oct 31 01:40:24.934154 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.934106    5650 generic.go:356] "Generic (PLEG): container finished" podID="148e0558-73c9-40e3-a412-594d1b9a6c72" containerID="5993c60c21282b2ee7b3a826877ab4209065fe9a81173c6624085a873f1238d6" exitCode=0
Oct 31 01:40:24.934694 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.934183    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq"
Oct 31 01:40:24.934694 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.934183    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq" event={"ID":"148e0558-73c9-40e3-a412-594d1b9a6c72","Type":"ContainerDied","Data":"5993c60c21282b2ee7b3a826877ab4209065fe9a81173c6624085a873f1238d6"}
Oct 31 01:40:24.934694 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.934264    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq" event={"ID":"148e0558-73c9-40e3-a412-594d1b9a6c72","Type":"ContainerDied","Data":"b78e64809c3de37ab281198c211e43bb5554efa7d946a8b60799b8c652f9494e"}
Oct 31 01:40:24.934694 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.934306    5650 scope.go:122] "RemoveContainer" containerID="5993c60c21282b2ee7b3a826877ab4209065fe9a81173c6624085a873f1238d6"
Oct 31 01:40:24.935680 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.935657    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:24.938014 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.937291    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:24.953695 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.953665    5650 scope.go:122] "RemoveContainer" containerID="5993c60c21282b2ee7b3a826877ab4209065fe9a81173c6624085a873f1238d6"
Oct 31 01:40:24.954212 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:40:24.954183    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5993c60c21282b2ee7b3a826877ab4209065fe9a81173c6624085a873f1238d6\": container with ID starting with 5993c60c21282b2ee7b3a826877ab4209065fe9a81173c6624085a873f1238d6 not found: ID does not exist" containerID="5993c60c21282b2ee7b3a826877ab4209065fe9a81173c6624085a873f1238d6"
Oct 31 01:40:24.954326 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.954227    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5993c60c21282b2ee7b3a826877ab4209065fe9a81173c6624085a873f1238d6"} err="failed to get container status \"5993c60c21282b2ee7b3a826877ab4209065fe9a81173c6624085a873f1238d6\": rpc error: code = NotFound desc = could not find container \"5993c60c21282b2ee7b3a826877ab4209065fe9a81173c6624085a873f1238d6\": container with ID starting with 5993c60c21282b2ee7b3a826877ab4209065fe9a81173c6624085a873f1238d6 not found: ID does not exist"
Oct 31 01:40:24.981958 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.981840    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw"
Oct 31 01:40:24.991144 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.991089    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"]
Oct 31 01:40:24.998103 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:24.998072    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq"]
Oct 31 01:40:25.000514 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:40:25.000480    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0ee471_3ebf_426f_88e8_689ddc123d8e.slice/crio-4ed865c42a4f2fca8fed7bda29887bd30f117133ae02b03c07b88aa3aebd411a WatchSource:0}: Error finding container 4ed865c42a4f2fca8fed7bda29887bd30f117133ae02b03c07b88aa3aebd411a: Status 404 returned error can't find the container with id 4ed865c42a4f2fca8fed7bda29887bd30f117133ae02b03c07b88aa3aebd411a
Oct 31 01:40:25.001664 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.001552    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4db52a49-7bbbd869d5-rl7rq"]
Oct 31 01:40:25.002706 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.002670    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:40:25.002829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.002758    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:40:25.002829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.002802    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:40:25.028694 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.028660    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p"
Oct 31 01:40:25.277605 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.277514    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m" podUID="4aebe63d-3854-4222-ad9c-11a6e5c93219" containerName="payload-processing" containerID="cri-o://5b58f100694b550ec026d436a199414c2a02c68f11d4fd309c59080c18440ce9" gracePeriod=25
Oct 31 01:40:25.284155 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.284126    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw"]
Oct 31 01:40:25.288192 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:40:25.288162    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f0ddb1_3758_4a14_a2e2_2c7a928b3564.slice/crio-b6cd9586cc5f6a2d4b3cbd14af686d3597b9bc7e3df712dcdf85a137dc3f47c7 WatchSource:0}: Error finding container b6cd9586cc5f6a2d4b3cbd14af686d3597b9bc7e3df712dcdf85a137dc3f47c7: Status 404 returned error can't find the container with id b6cd9586cc5f6a2d4b3cbd14af686d3597b9bc7e3df712dcdf85a137dc3f47c7
Oct 31 01:40:25.302302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.302265    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8" podUID="7850ced7-5d83-4d43-84e9-c88400db6291" containerName="payload-pre-processing" containerID="cri-o://9bdd463555093d509fd8ab9fd69a6e559af372fce566f3d909d0792fc7efc608" gracePeriod=25
Oct 31 01:40:25.340340 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.340306    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p"]
Oct 31 01:40:25.352070 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:40:25.351989    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e45eafd_ca74_4a6e_b2c7_e3fd8cd15d5e.slice/crio-5d215e0797b6ea88bb95117911e3ed0cdeb597518a80d533b8db5293445831d7 WatchSource:0}: Error finding container 5d215e0797b6ea88bb95117911e3ed0cdeb597518a80d533b8db5293445831d7: Status 404 returned error can't find the container with id 5d215e0797b6ea88bb95117911e3ed0cdeb597518a80d533b8db5293445831d7
Oct 31 01:40:25.542125 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.542008    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx" podUID="30c67387-a038-4a9c-a615-7989542d2b6b" containerName="payload-processing" containerID="cri-o://3a06acb7ba2162c862e058c0619ab983fe647c7c84f6dced515948fc6f6f738d" gracePeriod=25
Oct 31 01:40:25.557731 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.557675    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs" podUID="0616938c-466b-41eb-ac2b-9e66fa2bfa46" containerName="payload-pre-processing" containerID="cri-o://38f90fcd6896b9443ee29a05b041e0e094616291c79291293e276e21b4eff345" gracePeriod=25
Oct 31 01:40:25.671526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.671494    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m"
Oct 31 01:40:25.681906 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.676321    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8"
Oct 31 01:40:25.785016 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.784974    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7850ced7-5d83-4d43-84e9-c88400db6291-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7850ced7-5d83-4d43-84e9-c88400db6291-plugins-config-volume\") pod \"7850ced7-5d83-4d43-84e9-c88400db6291\" (UID: \"7850ced7-5d83-4d43-84e9-c88400db6291\") "
Oct 31 01:40:25.785180 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.785062    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7850ced7-5d83-4d43-84e9-c88400db6291-kube-api-access-mnqlx\" (UniqueName: \"kubernetes.io/projected/7850ced7-5d83-4d43-84e9-c88400db6291-kube-api-access-mnqlx\") pod \"7850ced7-5d83-4d43-84e9-c88400db6291\" (UID: \"7850ced7-5d83-4d43-84e9-c88400db6291\") "
Oct 31 01:40:25.785180 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.785128    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aebe63d-3854-4222-ad9c-11a6e5c93219-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aebe63d-3854-4222-ad9c-11a6e5c93219-plugins-config-volume\") pod \"4aebe63d-3854-4222-ad9c-11a6e5c93219\" (UID: \"4aebe63d-3854-4222-ad9c-11a6e5c93219\") "
Oct 31 01:40:25.785301 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.785280    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aebe63d-3854-4222-ad9c-11a6e5c93219-kube-api-access-26fsd\" (UniqueName: \"kubernetes.io/projected/4aebe63d-3854-4222-ad9c-11a6e5c93219-kube-api-access-26fsd\") pod \"4aebe63d-3854-4222-ad9c-11a6e5c93219\" (UID: \"4aebe63d-3854-4222-ad9c-11a6e5c93219\") "
Oct 31 01:40:25.787521 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.787486    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aebe63d-3854-4222-ad9c-11a6e5c93219-plugins-config-volume" pod "4aebe63d-3854-4222-ad9c-11a6e5c93219" (UID: "4aebe63d-3854-4222-ad9c-11a6e5c93219"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:25.787641 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.787563    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7850ced7-5d83-4d43-84e9-c88400db6291-plugins-config-volume" pod "7850ced7-5d83-4d43-84e9-c88400db6291" (UID: "7850ced7-5d83-4d43-84e9-c88400db6291"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:25.788071 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.788045    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aebe63d-3854-4222-ad9c-11a6e5c93219-kube-api-access-26fsd" pod "4aebe63d-3854-4222-ad9c-11a6e5c93219" (UID: "4aebe63d-3854-4222-ad9c-11a6e5c93219"). InnerVolumeSpecName "kube-api-access-26fsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:25.789877 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.789852    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7850ced7-5d83-4d43-84e9-c88400db6291-kube-api-access-mnqlx" pod "7850ced7-5d83-4d43-84e9-c88400db6291" (UID: "7850ced7-5d83-4d43-84e9-c88400db6291"). InnerVolumeSpecName "kube-api-access-mnqlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:25.843353 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.843322    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx"
Oct 31 01:40:25.850838 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.850807    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs"
Oct 31 01:40:25.887093 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.887019    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7850ced7-5d83-4d43-84e9-c88400db6291-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:25.887093 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.887059    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnqlx\" (UniqueName: \"kubernetes.io/projected/7850ced7-5d83-4d43-84e9-c88400db6291-kube-api-access-mnqlx\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:25.887093 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.887077    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aebe63d-3854-4222-ad9c-11a6e5c93219-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:25.887093 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.887091    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26fsd\" (UniqueName: \"kubernetes.io/projected/4aebe63d-3854-4222-ad9c-11a6e5c93219-kube-api-access-26fsd\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:25.941810 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.941777    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw" event={"ID":"35f0ddb1-3758-4a14-a2e2-2c7a928b3564","Type":"ContainerStarted","Data":"97d4a78fcd99709176951137a6a5e07e325bafa066bb7663b8021cd346be6e69"}
Oct 31 01:40:25.942251 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.941828    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw" event={"ID":"35f0ddb1-3758-4a14-a2e2-2c7a928b3564","Type":"ContainerStarted","Data":"b6cd9586cc5f6a2d4b3cbd14af686d3597b9bc7e3df712dcdf85a137dc3f47c7"}
Oct 31 01:40:25.942251 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.941902    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw"
Oct 31 01:40:25.943844 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.943817    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8" event={"ID":"af0ee471-3ebf-426f-88e8-689ddc123d8e","Type":"ContainerStarted","Data":"fe2949ea3820d9e9e31cc129190a905dc7b124e5b6c3d8528eac8f17423881f1"}
Oct 31 01:40:25.943990 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.943865    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8" event={"ID":"af0ee471-3ebf-426f-88e8-689ddc123d8e","Type":"ContainerStarted","Data":"4ed865c42a4f2fca8fed7bda29887bd30f117133ae02b03c07b88aa3aebd411a"}
Oct 31 01:40:25.945685 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.945658    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p" event={"ID":"7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e","Type":"ContainerStarted","Data":"b07ae836aa64d66b7686b5255f24807f93e13ddb1686c70b9f39d6e698af38ad"}
Oct 31 01:40:25.945831 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.945702    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p" event={"ID":"7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e","Type":"ContainerStarted","Data":"5d215e0797b6ea88bb95117911e3ed0cdeb597518a80d533b8db5293445831d7"}
Oct 31 01:40:25.945831 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.945759    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p"
Oct 31 01:40:25.947093 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.947071    5650 generic.go:356] "Generic (PLEG): container finished" podID="7850ced7-5d83-4d43-84e9-c88400db6291" containerID="9bdd463555093d509fd8ab9fd69a6e559af372fce566f3d909d0792fc7efc608" exitCode=0
Oct 31 01:40:25.947200 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.947104    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8"
Oct 31 01:40:25.947200 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.947158    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8" event={"ID":"7850ced7-5d83-4d43-84e9-c88400db6291","Type":"ContainerDied","Data":"9bdd463555093d509fd8ab9fd69a6e559af372fce566f3d909d0792fc7efc608"}
Oct 31 01:40:25.947318 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.947209    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8" event={"ID":"7850ced7-5d83-4d43-84e9-c88400db6291","Type":"ContainerDied","Data":"b10ac224e81c26bf938fc13b45c672eea849a844c779bf33f9b7a9a1db540799"}
Oct 31 01:40:25.947318 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.947241    5650 scope.go:122] "RemoveContainer" containerID="9bdd463555093d509fd8ab9fd69a6e559af372fce566f3d909d0792fc7efc608"
Oct 31 01:40:25.949540 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.949520    5650 generic.go:356] "Generic (PLEG): container finished" podID="4aebe63d-3854-4222-ad9c-11a6e5c93219" containerID="5b58f100694b550ec026d436a199414c2a02c68f11d4fd309c59080c18440ce9" exitCode=0
Oct 31 01:40:25.949707 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.949563    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m"
Oct 31 01:40:25.949707 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.949589    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m" event={"ID":"4aebe63d-3854-4222-ad9c-11a6e5c93219","Type":"ContainerDied","Data":"5b58f100694b550ec026d436a199414c2a02c68f11d4fd309c59080c18440ce9"}
Oct 31 01:40:25.949707 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.949626    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m" event={"ID":"4aebe63d-3854-4222-ad9c-11a6e5c93219","Type":"ContainerDied","Data":"3b58fe18b2641cb8ff66788de5b7503fa76e3b1685163e20bf86d9543566a71c"}
Oct 31 01:40:25.951100 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.951075    5650 generic.go:356] "Generic (PLEG): container finished" podID="0616938c-466b-41eb-ac2b-9e66fa2bfa46" containerID="38f90fcd6896b9443ee29a05b041e0e094616291c79291293e276e21b4eff345" exitCode=0
Oct 31 01:40:25.951213 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.951110    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs"
Oct 31 01:40:25.951213 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.951157    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs" event={"ID":"0616938c-466b-41eb-ac2b-9e66fa2bfa46","Type":"ContainerDied","Data":"38f90fcd6896b9443ee29a05b041e0e094616291c79291293e276e21b4eff345"}
Oct 31 01:40:25.951213 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.951203    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs" event={"ID":"0616938c-466b-41eb-ac2b-9e66fa2bfa46","Type":"ContainerDied","Data":"b353b3c8bfa1ff4055d4d0c66008ebd9e386cb8cfa61be55d843faa3ea327626"}
Oct 31 01:40:25.952683 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.952530    5650 generic.go:356] "Generic (PLEG): container finished" podID="30c67387-a038-4a9c-a615-7989542d2b6b" containerID="3a06acb7ba2162c862e058c0619ab983fe647c7c84f6dced515948fc6f6f738d" exitCode=0
Oct 31 01:40:25.952683 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.952569    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx"
Oct 31 01:40:25.952683 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.952593    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx" event={"ID":"30c67387-a038-4a9c-a615-7989542d2b6b","Type":"ContainerDied","Data":"3a06acb7ba2162c862e058c0619ab983fe647c7c84f6dced515948fc6f6f738d"}
Oct 31 01:40:25.952683 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.952624    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx" event={"ID":"30c67387-a038-4a9c-a615-7989542d2b6b","Type":"ContainerDied","Data":"7cf28822f352aed93c07addcaab9d52f87379a4cce42e01c3e9a52ae2d8c2b7a"}
Oct 31 01:40:25.959573 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.959543    5650 scope.go:122] "RemoveContainer" containerID="9bdd463555093d509fd8ab9fd69a6e559af372fce566f3d909d0792fc7efc608"
Oct 31 01:40:25.959904 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:40:25.959877    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bdd463555093d509fd8ab9fd69a6e559af372fce566f3d909d0792fc7efc608\": container with ID starting with 9bdd463555093d509fd8ab9fd69a6e559af372fce566f3d909d0792fc7efc608 not found: ID does not exist" containerID="9bdd463555093d509fd8ab9fd69a6e559af372fce566f3d909d0792fc7efc608"
Oct 31 01:40:25.959984 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.959916    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bdd463555093d509fd8ab9fd69a6e559af372fce566f3d909d0792fc7efc608"} err="failed to get container status \"9bdd463555093d509fd8ab9fd69a6e559af372fce566f3d909d0792fc7efc608\": rpc error: code = NotFound desc = could not find container \"9bdd463555093d509fd8ab9fd69a6e559af372fce566f3d909d0792fc7efc608\": container with ID starting with 9bdd463555093d509fd8ab9fd69a6e559af372fce566f3d909d0792fc7efc608 not found: ID does not exist"
Oct 31 01:40:25.959984 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.959938    5650 scope.go:122] "RemoveContainer" containerID="5b58f100694b550ec026d436a199414c2a02c68f11d4fd309c59080c18440ce9"
Oct 31 01:40:25.963651 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.963609    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw" podStartSLOduration=1.963596243 podStartE2EDuration="1.963596243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:25.961834243 +0000 UTC m=+1181.965548535" watchObservedRunningTime="2026-10-31 01:40:25.963596243 +0000 UTC m=+1181.967310533"
Oct 31 01:40:25.970717 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.970691    5650 scope.go:122] "RemoveContainer" containerID="5b58f100694b550ec026d436a199414c2a02c68f11d4fd309c59080c18440ce9"
Oct 31 01:40:25.971044 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:40:25.971018    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b58f100694b550ec026d436a199414c2a02c68f11d4fd309c59080c18440ce9\": container with ID starting with 5b58f100694b550ec026d436a199414c2a02c68f11d4fd309c59080c18440ce9 not found: ID does not exist" containerID="5b58f100694b550ec026d436a199414c2a02c68f11d4fd309c59080c18440ce9"
Oct 31 01:40:25.971117 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.971055    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b58f100694b550ec026d436a199414c2a02c68f11d4fd309c59080c18440ce9"} err="failed to get container status \"5b58f100694b550ec026d436a199414c2a02c68f11d4fd309c59080c18440ce9\": rpc error: code = NotFound desc = could not find container \"5b58f100694b550ec026d436a199414c2a02c68f11d4fd309c59080c18440ce9\": container with ID starting with 5b58f100694b550ec026d436a199414c2a02c68f11d4fd309c59080c18440ce9 not found: ID does not exist"
Oct 31 01:40:25.971117 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.971078    5650 scope.go:122] "RemoveContainer" containerID="38f90fcd6896b9443ee29a05b041e0e094616291c79291293e276e21b4eff345"
Oct 31 01:40:25.980281 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.980255    5650 scope.go:122] "RemoveContainer" containerID="38f90fcd6896b9443ee29a05b041e0e094616291c79291293e276e21b4eff345"
Oct 31 01:40:25.980662 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:40:25.980638    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38f90fcd6896b9443ee29a05b041e0e094616291c79291293e276e21b4eff345\": container with ID starting with 38f90fcd6896b9443ee29a05b041e0e094616291c79291293e276e21b4eff345 not found: ID does not exist" containerID="38f90fcd6896b9443ee29a05b041e0e094616291c79291293e276e21b4eff345"
Oct 31 01:40:25.980766 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.980673    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38f90fcd6896b9443ee29a05b041e0e094616291c79291293e276e21b4eff345"} err="failed to get container status \"38f90fcd6896b9443ee29a05b041e0e094616291c79291293e276e21b4eff345\": rpc error: code = NotFound desc = could not find container \"38f90fcd6896b9443ee29a05b041e0e094616291c79291293e276e21b4eff345\": container with ID starting with 38f90fcd6896b9443ee29a05b041e0e094616291c79291293e276e21b4eff345 not found: ID does not exist"
Oct 31 01:40:25.980766 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.980699    5650 scope.go:122] "RemoveContainer" containerID="3a06acb7ba2162c862e058c0619ab983fe647c7c84f6dced515948fc6f6f738d"
Oct 31 01:40:25.988091 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.988066    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30c67387-a038-4a9c-a615-7989542d2b6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30c67387-a038-4a9c-a615-7989542d2b6b-plugins-config-volume\") pod \"30c67387-a038-4a9c-a615-7989542d2b6b\" (UID: \"30c67387-a038-4a9c-a615-7989542d2b6b\") "
Oct 31 01:40:25.988202 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.988121    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c67387-a038-4a9c-a615-7989542d2b6b-kube-api-access-vtv9v\" (UniqueName: \"kubernetes.io/projected/30c67387-a038-4a9c-a615-7989542d2b6b-kube-api-access-vtv9v\") pod \"30c67387-a038-4a9c-a615-7989542d2b6b\" (UID: \"30c67387-a038-4a9c-a615-7989542d2b6b\") "
Oct 31 01:40:25.988202 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.988175    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0616938c-466b-41eb-ac2b-9e66fa2bfa46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0616938c-466b-41eb-ac2b-9e66fa2bfa46-plugins-config-volume\") pod \"0616938c-466b-41eb-ac2b-9e66fa2bfa46\" (UID: \"0616938c-466b-41eb-ac2b-9e66fa2bfa46\") "
Oct 31 01:40:25.988364 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.988350    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0616938c-466b-41eb-ac2b-9e66fa2bfa46-kube-api-access-n9f9q\" (UniqueName: \"kubernetes.io/projected/0616938c-466b-41eb-ac2b-9e66fa2bfa46-kube-api-access-n9f9q\") pod \"0616938c-466b-41eb-ac2b-9e66fa2bfa46\" (UID: \"0616938c-466b-41eb-ac2b-9e66fa2bfa46\") "
Oct 31 01:40:25.988533 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.988507    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c67387-a038-4a9c-a615-7989542d2b6b-plugins-config-volume" pod "30c67387-a038-4a9c-a615-7989542d2b6b" (UID: "30c67387-a038-4a9c-a615-7989542d2b6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:25.988604 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.988534    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0616938c-466b-41eb-ac2b-9e66fa2bfa46-plugins-config-volume" pod "0616938c-466b-41eb-ac2b-9e66fa2bfa46" (UID: "0616938c-466b-41eb-ac2b-9e66fa2bfa46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:25.988718 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.988702    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30c67387-a038-4a9c-a615-7989542d2b6b-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:25.988764 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.988725    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0616938c-466b-41eb-ac2b-9e66fa2bfa46-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:25.990015 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.989994    5650 scope.go:122] "RemoveContainer" containerID="3a06acb7ba2162c862e058c0619ab983fe647c7c84f6dced515948fc6f6f738d"
Oct 31 01:40:25.990369 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:40:25.990344    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a06acb7ba2162c862e058c0619ab983fe647c7c84f6dced515948fc6f6f738d\": container with ID starting with 3a06acb7ba2162c862e058c0619ab983fe647c7c84f6dced515948fc6f6f738d not found: ID does not exist" containerID="3a06acb7ba2162c862e058c0619ab983fe647c7c84f6dced515948fc6f6f738d"
Oct 31 01:40:25.990522 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.990378    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a06acb7ba2162c862e058c0619ab983fe647c7c84f6dced515948fc6f6f738d"} err="failed to get container status \"3a06acb7ba2162c862e058c0619ab983fe647c7c84f6dced515948fc6f6f738d\": rpc error: code = NotFound desc = could not find container \"3a06acb7ba2162c862e058c0619ab983fe647c7c84f6dced515948fc6f6f738d\": container with ID starting with 3a06acb7ba2162c862e058c0619ab983fe647c7c84f6dced515948fc6f6f738d not found: ID does not exist"
Oct 31 01:40:25.990658 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.990635    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0616938c-466b-41eb-ac2b-9e66fa2bfa46-kube-api-access-n9f9q" pod "0616938c-466b-41eb-ac2b-9e66fa2bfa46" (UID: "0616938c-466b-41eb-ac2b-9e66fa2bfa46"). InnerVolumeSpecName "kube-api-access-n9f9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:25.990726 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.990708    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c67387-a038-4a9c-a615-7989542d2b6b-kube-api-access-vtv9v" pod "30c67387-a038-4a9c-a615-7989542d2b6b" (UID: "30c67387-a038-4a9c-a615-7989542d2b6b"). InnerVolumeSpecName "kube-api-access-vtv9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:25.998026 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:25.997982    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8" podStartSLOduration=1.997970121 podStartE2EDuration="1.997970121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:25.996143713 +0000 UTC m=+1181.999858003" watchObservedRunningTime="2026-10-31 01:40:25.997970121 +0000 UTC m=+1182.001684412"
Oct 31 01:40:26.012543 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.012511    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8"]
Oct 31 01:40:26.019329 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.019298    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4bf7efc6-5494979667-c2dg8"]
Oct 31 01:40:26.038886 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.038832    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p" podStartSLOduration=2.03881744 podStartE2EDuration="2.03881744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:26.038160454 +0000 UTC m=+1182.041874747" watchObservedRunningTime="2026-10-31 01:40:26.03881744 +0000 UTC m=+1182.042531730"
Oct 31 01:40:26.060404 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.060366    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m"]
Oct 31 01:40:26.066796 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.066764    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4bf7efc6-54f688bc7-m4r2m"]
Oct 31 01:40:26.090630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.090551    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtv9v\" (UniqueName: \"kubernetes.io/projected/30c67387-a038-4a9c-a615-7989542d2b6b-kube-api-access-vtv9v\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:26.090630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.090584    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9f9q\" (UniqueName: \"kubernetes.io/projected/0616938c-466b-41eb-ac2b-9e66fa2bfa46-kube-api-access-n9f9q\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:26.124079 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.124043    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="148e0558-73c9-40e3-a412-594d1b9a6c72" path="/var/lib/kubelet/pods/148e0558-73c9-40e3-a412-594d1b9a6c72/volumes"
Oct 31 01:40:26.124579 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.124563    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aebe63d-3854-4222-ad9c-11a6e5c93219" path="/var/lib/kubelet/pods/4aebe63d-3854-4222-ad9c-11a6e5c93219/volumes"
Oct 31 01:40:26.125028 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.125013    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7850ced7-5d83-4d43-84e9-c88400db6291" path="/var/lib/kubelet/pods/7850ced7-5d83-4d43-84e9-c88400db6291/volumes"
Oct 31 01:40:26.125989 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.125968    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5669b53-1a4d-4b27-8088-1f36f1842466" path="/var/lib/kubelet/pods/b5669b53-1a4d-4b27-8088-1f36f1842466/volumes"
Oct 31 01:40:26.166542 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.166511    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"]
Oct 31 01:40:26.167010 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.166988    5650 cpu_manager.go:414] "containerMap: removing container" podUID="4aebe63d-3854-4222-ad9c-11a6e5c93219" containerName="payload-processing"
Oct 31 01:40:26.167010 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.167007    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aebe63d-3854-4222-ad9c-11a6e5c93219" containerName="payload-processing"
Oct 31 01:40:26.167206 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.167045    5650 cpu_manager.go:414] "containerMap: removing container" podUID="0616938c-466b-41eb-ac2b-9e66fa2bfa46" containerName="payload-pre-processing"
Oct 31 01:40:26.167206 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.167052    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0616938c-466b-41eb-ac2b-9e66fa2bfa46" containerName="payload-pre-processing"
Oct 31 01:40:26.167206 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.167066    5650 cpu_manager.go:414] "containerMap: removing container" podUID="7850ced7-5d83-4d43-84e9-c88400db6291" containerName="payload-pre-processing"
Oct 31 01:40:26.167206 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.167072    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7850ced7-5d83-4d43-84e9-c88400db6291" containerName="payload-pre-processing"
Oct 31 01:40:26.167206 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.167090    5650 cpu_manager.go:414] "containerMap: removing container" podUID="30c67387-a038-4a9c-a615-7989542d2b6b" containerName="payload-processing"
Oct 31 01:40:26.167206 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.167095    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c67387-a038-4a9c-a615-7989542d2b6b" containerName="payload-processing"
Oct 31 01:40:26.167206 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.167165    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="0616938c-466b-41eb-ac2b-9e66fa2bfa46" containerName="payload-pre-processing"
Oct 31 01:40:26.167206 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.167173    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="7850ced7-5d83-4d43-84e9-c88400db6291" containerName="payload-pre-processing"
Oct 31 01:40:26.167206 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.167183    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c67387-a038-4a9c-a615-7989542d2b6b" containerName="payload-processing"
Oct 31 01:40:26.167206 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.167189    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aebe63d-3854-4222-ad9c-11a6e5c93219" containerName="payload-processing"
Oct 31 01:40:26.172316 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.172291    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.184853 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.184558    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-3cf6f4-openshift-default-dockercfg-5t5rj\""
Oct 31 01:40:26.186369 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.186338    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"]
Oct 31 01:40:26.272701 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.272664    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs"]
Oct 31 01:40:26.274563 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.274544    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-aba34286-864fbc8f99-hknhs"]
Oct 31 01:40:26.292626 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.292588    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.292796 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.292674    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.292796 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.292729    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.292796 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.292779    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8ad70f-aced-426e-8d7a-d1c17c256582-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.292899 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.292802    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg9lc\" (UniqueName: \"kubernetes.io/projected/7f8ad70f-aced-426e-8d7a-d1c17c256582-kube-api-access-fg9lc\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.292899 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.292870    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.292964 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.292895    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.292964 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.292923    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.293022 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.292964    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.293673 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.293652    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx"]
Oct 31 01:40:26.300689 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.300659    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-aba34286-7475c7489f-hfzqx"]
Oct 31 01:40:26.394205 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.394113    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.394205 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.394164    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8ad70f-aced-426e-8d7a-d1c17c256582-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.394205 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.394192    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg9lc\" (UniqueName: \"kubernetes.io/projected/7f8ad70f-aced-426e-8d7a-d1c17c256582-kube-api-access-fg9lc\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.394532 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.394262    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.394532 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.394287    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.394532 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.394312    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.394715 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.394573    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.394715 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.394649    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.394819 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.394749    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.394875 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.394857    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.394987 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.394958    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8ad70f-aced-426e-8d7a-d1c17c256582-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.395062 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.395003    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.395130 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.395080    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.395431 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.395404    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.397020 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.396993    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.397166 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.397150    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.408360 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.408329    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg9lc\" (UniqueName: \"kubernetes.io/projected/7f8ad70f-aced-426e-8d7a-d1c17c256582-kube-api-access-fg9lc\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.408655 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.408636    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.487308 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.487267    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:26.684658 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.684571    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:26.690066 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.690038    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:26.772796 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.772771    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"]
Oct 31 01:40:26.774788 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:40:26.774762    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8ad70f_aced_426e_8d7a_d1c17c256582.slice/crio-c35ffc3231b773e3986a4a85b08367d812cabb126f1d1e3c3d6121cfd059f9a9 WatchSource:0}: Error finding container c35ffc3231b773e3986a4a85b08367d812cabb126f1d1e3c3d6121cfd059f9a9: Status 404 returned error can't find the container with id c35ffc3231b773e3986a4a85b08367d812cabb126f1d1e3c3d6121cfd059f9a9
Oct 31 01:40:26.777227 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.777197    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:40:26.777297 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.777262    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:40:26.777329 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.777295    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:40:26.961780 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.961739    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv" event={"ID":"7f8ad70f-aced-426e-8d7a-d1c17c256582","Type":"ContainerStarted","Data":"b564f738c4a77bb8f71c143af141ec5494444da818473d63c951295aa1db2580"}
Oct 31 01:40:26.962388 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.961805    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv" event={"ID":"7f8ad70f-aced-426e-8d7a-d1c17c256582","Type":"ContainerStarted","Data":"c35ffc3231b773e3986a4a85b08367d812cabb126f1d1e3c3d6121cfd059f9a9"}
Oct 31 01:40:26.966578 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.966539    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:26.967212 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.967185    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:26.994514 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:26.994438    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv" podStartSLOduration=0.99442139 podStartE2EDuration="994.42139ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:26.992531557 +0000 UTC m=+1182.996245847" watchObservedRunningTime="2026-10-31 01:40:26.99442139 +0000 UTC m=+1182.998135681"
Oct 31 01:40:27.440000 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:27.439899    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9" podUID="0274a049-314d-4de4-8424-ca88f05ee373" containerName="payload-pre-processing" containerID="cri-o://7769f175eaba3dc4062044fd141f74eccedc9a0aa0b8a6cfd6f4b5ebdd286d84" gracePeriod=25
Oct 31 01:40:27.487699 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:27.487658    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:27.493366 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:27.493334    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:27.705428 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:27.705398    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9"
Oct 31 01:40:27.812746 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:27.812702    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0274a049-314d-4de4-8424-ca88f05ee373-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0274a049-314d-4de4-8424-ca88f05ee373-plugins-config-volume\") pod \"0274a049-314d-4de4-8424-ca88f05ee373\" (UID: \"0274a049-314d-4de4-8424-ca88f05ee373\") "
Oct 31 01:40:27.812930 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:27.812769    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0274a049-314d-4de4-8424-ca88f05ee373-kube-api-access-vhnk8\" (UniqueName: \"kubernetes.io/projected/0274a049-314d-4de4-8424-ca88f05ee373-kube-api-access-vhnk8\") pod \"0274a049-314d-4de4-8424-ca88f05ee373\" (UID: \"0274a049-314d-4de4-8424-ca88f05ee373\") "
Oct 31 01:40:27.813140 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:27.813117    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0274a049-314d-4de4-8424-ca88f05ee373-plugins-config-volume" pod "0274a049-314d-4de4-8424-ca88f05ee373" (UID: "0274a049-314d-4de4-8424-ca88f05ee373"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:27.814993 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:27.814960    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0274a049-314d-4de4-8424-ca88f05ee373-kube-api-access-vhnk8" pod "0274a049-314d-4de4-8424-ca88f05ee373" (UID: "0274a049-314d-4de4-8424-ca88f05ee373"). InnerVolumeSpecName "kube-api-access-vhnk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:27.914035 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:27.913957    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0274a049-314d-4de4-8424-ca88f05ee373-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:27.914035 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:27.913989    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhnk8\" (UniqueName: \"kubernetes.io/projected/0274a049-314d-4de4-8424-ca88f05ee373-kube-api-access-vhnk8\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:27.977256 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:27.977145    5650 generic.go:356] "Generic (PLEG): container finished" podID="0274a049-314d-4de4-8424-ca88f05ee373" containerID="7769f175eaba3dc4062044fd141f74eccedc9a0aa0b8a6cfd6f4b5ebdd286d84" exitCode=0
Oct 31 01:40:27.977256 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:27.977212    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9"
Oct 31 01:40:27.977820 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:27.977250    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9" event={"ID":"0274a049-314d-4de4-8424-ca88f05ee373","Type":"ContainerDied","Data":"7769f175eaba3dc4062044fd141f74eccedc9a0aa0b8a6cfd6f4b5ebdd286d84"}
Oct 31 01:40:27.977820 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:27.977317    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9" event={"ID":"0274a049-314d-4de4-8424-ca88f05ee373","Type":"ContainerDied","Data":"98352dec7ded8b0e86e64e3b28752a0e3757a2384fc4ec9bedb497af760076c5"}
Oct 31 01:40:27.977820 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:27.977350    5650 scope.go:122] "RemoveContainer" containerID="7769f175eaba3dc4062044fd141f74eccedc9a0aa0b8a6cfd6f4b5ebdd286d84"
Oct 31 01:40:27.981803 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:27.981767    5650 generic.go:356] "Generic (PLEG): container finished" podID="a85900b2-fa9d-42d9-bd0f-70425dbcfda7" containerID="0b9a295e0870920001f221d7e103023a85549b407987fd9d385c83cb66eea6c8" exitCode=0
Oct 31 01:40:27.982143 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:27.981791    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg" event={"ID":"a85900b2-fa9d-42d9-bd0f-70425dbcfda7","Type":"ContainerDied","Data":"0b9a295e0870920001f221d7e103023a85549b407987fd9d385c83cb66eea6c8"}
Oct 31 01:40:27.982682 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:27.982653    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:27.983764 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:27.983742    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:27.989695 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:27.989670    5650 scope.go:122] "RemoveContainer" containerID="7769f175eaba3dc4062044fd141f74eccedc9a0aa0b8a6cfd6f4b5ebdd286d84"
Oct 31 01:40:27.990070 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:40:27.990052    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7769f175eaba3dc4062044fd141f74eccedc9a0aa0b8a6cfd6f4b5ebdd286d84\": container with ID starting with 7769f175eaba3dc4062044fd141f74eccedc9a0aa0b8a6cfd6f4b5ebdd286d84 not found: ID does not exist" containerID="7769f175eaba3dc4062044fd141f74eccedc9a0aa0b8a6cfd6f4b5ebdd286d84"
Oct 31 01:40:27.990131 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:27.990079    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7769f175eaba3dc4062044fd141f74eccedc9a0aa0b8a6cfd6f4b5ebdd286d84"} err="failed to get container status \"7769f175eaba3dc4062044fd141f74eccedc9a0aa0b8a6cfd6f4b5ebdd286d84\": rpc error: code = NotFound desc = could not find container \"7769f175eaba3dc4062044fd141f74eccedc9a0aa0b8a6cfd6f4b5ebdd286d84\": container with ID starting with 7769f175eaba3dc4062044fd141f74eccedc9a0aa0b8a6cfd6f4b5ebdd286d84 not found: ID does not exist"
Oct 31 01:40:27.990131 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:27.990098    5650 scope.go:122] "RemoveContainer" containerID="f5abb7c6c0565747c41cc72a9d0028c055b042db194064ca6b7f614209cb33ca"
Oct 31 01:40:28.000341 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.000272    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:40:28.039031 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.038992    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9"]
Oct 31 01:40:28.047339 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.047303    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6198e156-56b455c78c-9wmm9"]
Oct 31 01:40:28.115079 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.115041    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-envoy\") pod \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") "
Oct 31 01:40:28.115274 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.115155    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-podinfo\") pod \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") "
Oct 31 01:40:28.115274 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.115192    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-workload-certs\") pod \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") "
Oct 31 01:40:28.115274 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.115241    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-data\") pod \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") "
Oct 31 01:40:28.115474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.115306    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-token\" (UniqueName: \"kubernetes.io/projected/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-token\") pod \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") "
Oct 31 01:40:28.115474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.115350    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istiod-ca-cert\") pod \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") "
Oct 31 01:40:28.115474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.115400    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-kube-api-access-bjt5z\" (UniqueName: \"kubernetes.io/projected/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-kube-api-access-bjt5z\") pod \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") "
Oct 31 01:40:28.115636 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.115492    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-workload-socket\") pod \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") "
Oct 31 01:40:28.115636 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.115528    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-credential-socket\") pod \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\" (UID: \"a85900b2-fa9d-42d9-bd0f-70425dbcfda7\") "
Oct 31 01:40:28.115636 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.115555    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-workload-certs" pod "a85900b2-fa9d-42d9-bd0f-70425dbcfda7" (UID: "a85900b2-fa9d-42d9-bd0f-70425dbcfda7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:28.115901 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.115876    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istiod-ca-cert" pod "a85900b2-fa9d-42d9-bd0f-70425dbcfda7" (UID: "a85900b2-fa9d-42d9-bd0f-70425dbcfda7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:28.116132 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.116113    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-credential-socket" pod "a85900b2-fa9d-42d9-bd0f-70425dbcfda7" (UID: "a85900b2-fa9d-42d9-bd0f-70425dbcfda7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:28.116216 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.116130    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-data" pod "a85900b2-fa9d-42d9-bd0f-70425dbcfda7" (UID: "a85900b2-fa9d-42d9-bd0f-70425dbcfda7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:28.116351 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.116324    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-workload-socket" pod "a85900b2-fa9d-42d9-bd0f-70425dbcfda7" (UID: "a85900b2-fa9d-42d9-bd0f-70425dbcfda7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:28.117046 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.117028    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:28.117152 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.117053    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:28.117835 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.117796    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-envoy" pod "a85900b2-fa9d-42d9-bd0f-70425dbcfda7" (UID: "a85900b2-fa9d-42d9-bd0f-70425dbcfda7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:28.117933 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.117908    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-podinfo" pod "a85900b2-fa9d-42d9-bd0f-70425dbcfda7" (UID: "a85900b2-fa9d-42d9-bd0f-70425dbcfda7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:40:28.118648 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.118621    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0274a049-314d-4de4-8424-ca88f05ee373" path="/var/lib/kubelet/pods/0274a049-314d-4de4-8424-ca88f05ee373/volumes"
Oct 31 01:40:28.118916 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.118889    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-token" pod "a85900b2-fa9d-42d9-bd0f-70425dbcfda7" (UID: "a85900b2-fa9d-42d9-bd0f-70425dbcfda7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:28.119070 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.119051    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-kube-api-access-bjt5z" pod "a85900b2-fa9d-42d9-bd0f-70425dbcfda7" (UID: "a85900b2-fa9d-42d9-bd0f-70425dbcfda7"). InnerVolumeSpecName "kube-api-access-bjt5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:28.119228 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.119210    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0616938c-466b-41eb-ac2b-9e66fa2bfa46" path="/var/lib/kubelet/pods/0616938c-466b-41eb-ac2b-9e66fa2bfa46/volumes"
Oct 31 01:40:28.119728 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.119711    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30c67387-a038-4a9c-a615-7989542d2b6b" path="/var/lib/kubelet/pods/30c67387-a038-4a9c-a615-7989542d2b6b/volumes"
Oct 31 01:40:28.160963 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:40:28.160925    5650 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85900b2_fa9d_42d9_bd0f_70425dbcfda7.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 01:40:28.218026 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.217988    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:28.218026 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.218020    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:28.218026 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.218029    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:28.218318 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.218038    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:28.218318 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.218047    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:28.218318 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.218056    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjt5z\" (UniqueName: \"kubernetes.io/projected/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-kube-api-access-bjt5z\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:28.218318 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.218070    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85900b2-fa9d-42d9-bd0f-70425dbcfda7-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:40:28.552557 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:40:28.552520    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:40:28.994468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.991285    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"
Oct 31 01:40:28.994468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.991302    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg" event={"ID":"a85900b2-fa9d-42d9-bd0f-70425dbcfda7","Type":"ContainerDied","Data":"0bc920952c426fbccf63d528260ceb5770e1a527c0f8d15d5c7771e917f57d67"}
Oct 31 01:40:28.994468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:28.991388    5650 scope.go:122] "RemoveContainer" containerID="0b9a295e0870920001f221d7e103023a85549b407987fd9d385c83cb66eea6c8"
Oct 31 01:40:29.058676 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.058634    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq"]
Oct 31 01:40:29.059140 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.059123    5650 cpu_manager.go:414] "containerMap: removing container" podUID="a85900b2-fa9d-42d9-bd0f-70425dbcfda7" containerName="istio-proxy"
Oct 31 01:40:29.059140 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.059138    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a85900b2-fa9d-42d9-bd0f-70425dbcfda7" containerName="istio-proxy"
Oct 31 01:40:29.059292 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.059149    5650 cpu_manager.go:414] "containerMap: removing container" podUID="0274a049-314d-4de4-8424-ca88f05ee373" containerName="payload-pre-processing"
Oct 31 01:40:29.059292 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.059154    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0274a049-314d-4de4-8424-ca88f05ee373" containerName="payload-pre-processing"
Oct 31 01:40:29.059292 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.059164    5650 cpu_manager.go:414] "containerMap: removing container" podUID="a85900b2-fa9d-42d9-bd0f-70425dbcfda7" containerName="istio-proxy"
Oct 31 01:40:29.059292 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.059169    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a85900b2-fa9d-42d9-bd0f-70425dbcfda7" containerName="istio-proxy"
Oct 31 01:40:29.059292 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.059265    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="a85900b2-fa9d-42d9-bd0f-70425dbcfda7" containerName="istio-proxy"
Oct 31 01:40:29.059292 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.059275    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="0274a049-314d-4de4-8424-ca88f05ee373" containerName="payload-pre-processing"
Oct 31 01:40:29.059292 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.059286    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="a85900b2-fa9d-42d9-bd0f-70425dbcfda7" containerName="istio-proxy"
Oct 31 01:40:29.064419 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.064392    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq"
Oct 31 01:40:29.083839 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.082812    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:40:29.083839 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.082915    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:40:29.089526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.087175    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"]
Oct 31 01:40:29.089526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.089383    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq"]
Oct 31 01:40:29.092815 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.092300    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-4db52a49-openshift-default-5fbc59957c-8q6xg"]
Oct 31 01:40:29.116222 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.113687    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-0e8b5c-serving-cert\""
Oct 31 01:40:29.116222 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.113804    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 01:40:29.116222 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.113873    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qjq8v\""
Oct 31 01:40:29.127608 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.127473    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0731e478-fead-48bf-9de2-4cf3ee6f184c-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq\" (UID: \"0731e478-fead-48bf-9de2-4cf3ee6f184c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq"
Oct 31 01:40:29.127608 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.127522    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkgmt\" (UniqueName: \"kubernetes.io/projected/0731e478-fead-48bf-9de2-4cf3ee6f184c-kube-api-access-lkgmt\") pod \"maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq\" (UID: \"0731e478-fead-48bf-9de2-4cf3ee6f184c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq"
Oct 31 01:40:29.128108 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.128071    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0731e478-fead-48bf-9de2-4cf3ee6f184c-metrics-certs\") pod \"maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq\" (UID: \"0731e478-fead-48bf-9de2-4cf3ee6f184c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq"
Oct 31 01:40:29.128562 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.128249    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0731e478-fead-48bf-9de2-4cf3ee6f184c-maas-api-tls\") pod \"maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq\" (UID: \"0731e478-fead-48bf-9de2-4cf3ee6f184c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq"
Oct 31 01:40:29.229183 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.229143    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0731e478-fead-48bf-9de2-4cf3ee6f184c-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq\" (UID: \"0731e478-fead-48bf-9de2-4cf3ee6f184c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq"
Oct 31 01:40:29.229383 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.229193    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkgmt\" (UniqueName: \"kubernetes.io/projected/0731e478-fead-48bf-9de2-4cf3ee6f184c-kube-api-access-lkgmt\") pod \"maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq\" (UID: \"0731e478-fead-48bf-9de2-4cf3ee6f184c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq"
Oct 31 01:40:29.229383 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.229251    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0731e478-fead-48bf-9de2-4cf3ee6f184c-metrics-certs\") pod \"maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq\" (UID: \"0731e478-fead-48bf-9de2-4cf3ee6f184c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq"
Oct 31 01:40:29.229383 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.229288    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0731e478-fead-48bf-9de2-4cf3ee6f184c-maas-api-tls\") pod \"maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq\" (UID: \"0731e478-fead-48bf-9de2-4cf3ee6f184c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq"
Oct 31 01:40:29.230192 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.230164    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0731e478-fead-48bf-9de2-4cf3ee6f184c-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq\" (UID: \"0731e478-fead-48bf-9de2-4cf3ee6f184c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq"
Oct 31 01:40:29.240084 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.240053    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0731e478-fead-48bf-9de2-4cf3ee6f184c-metrics-certs\") pod \"maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq\" (UID: \"0731e478-fead-48bf-9de2-4cf3ee6f184c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq"
Oct 31 01:40:29.241039 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.241015    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0731e478-fead-48bf-9de2-4cf3ee6f184c-maas-api-tls\") pod \"maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq\" (UID: \"0731e478-fead-48bf-9de2-4cf3ee6f184c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq"
Oct 31 01:40:29.242310 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.242282    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkgmt\" (UniqueName: \"kubernetes.io/projected/0731e478-fead-48bf-9de2-4cf3ee6f184c-kube-api-access-lkgmt\") pod \"maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq\" (UID: \"0731e478-fead-48bf-9de2-4cf3ee6f184c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq"
Oct 31 01:40:29.389328 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.389225    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq"
Oct 31 01:40:29.676775 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.676723    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq"]
Oct 31 01:40:29.690415 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:40:29.690380    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0731e478_fead_48bf_9de2_4cf3ee6f184c.slice/crio-a40e0f92b0a7de1f74c6a1b0c34b6ce1c239763168dedd724c568a170ac61bfe WatchSource:0}: Error finding container a40e0f92b0a7de1f74c6a1b0c34b6ce1c239763168dedd724c568a170ac61bfe: Status 404 returned error can't find the container with id a40e0f92b0a7de1f74c6a1b0c34b6ce1c239763168dedd724c568a170ac61bfe
Oct 31 01:40:29.998237 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:29.998199    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq" event={"ID":"0731e478-fead-48bf-9de2-4cf3ee6f184c","Type":"ContainerStarted","Data":"a40e0f92b0a7de1f74c6a1b0c34b6ce1c239763168dedd724c568a170ac61bfe"}
Oct 31 01:40:30.058735 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:30.058700    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn"]
Oct 31 01:40:30.063716 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:30.063693    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn"
Oct 31 01:40:30.071144 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:30.071101    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-7e5955\""
Oct 31 01:40:30.071328 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:30.071309    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-7e5955-dockercfg-g8f84\""
Oct 31 01:40:30.073986 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:30.073960    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn"]
Oct 31 01:40:30.119183 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:30.119146    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a85900b2-fa9d-42d9-bd0f-70425dbcfda7" path="/var/lib/kubelet/pods/a85900b2-fa9d-42d9-bd0f-70425dbcfda7/volumes"
Oct 31 01:40:30.138650 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:30.138605    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4640ac90-9987-4201-a94e-56a3102f0f0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn\" (UID: \"4640ac90-9987-4201-a94e-56a3102f0f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn"
Oct 31 01:40:30.138797 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:30.138666    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rttc\" (UniqueName: \"kubernetes.io/projected/4640ac90-9987-4201-a94e-56a3102f0f0e-kube-api-access-9rttc\") pod \"payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn\" (UID: \"4640ac90-9987-4201-a94e-56a3102f0f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn"
Oct 31 01:40:30.239602 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:30.239570    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4640ac90-9987-4201-a94e-56a3102f0f0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn\" (UID: \"4640ac90-9987-4201-a94e-56a3102f0f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn"
Oct 31 01:40:30.239717 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:30.239640    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rttc\" (UniqueName: \"kubernetes.io/projected/4640ac90-9987-4201-a94e-56a3102f0f0e-kube-api-access-9rttc\") pod \"payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn\" (UID: \"4640ac90-9987-4201-a94e-56a3102f0f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn"
Oct 31 01:40:30.240303 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:30.240284    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4640ac90-9987-4201-a94e-56a3102f0f0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn\" (UID: \"4640ac90-9987-4201-a94e-56a3102f0f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn"
Oct 31 01:40:30.252806 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:30.252782    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rttc\" (UniqueName: \"kubernetes.io/projected/4640ac90-9987-4201-a94e-56a3102f0f0e-kube-api-access-9rttc\") pod \"payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn\" (UID: \"4640ac90-9987-4201-a94e-56a3102f0f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn"
Oct 31 01:40:30.380111 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:30.380075    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn"
Oct 31 01:40:30.666927 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:30.666884    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn"]
Oct 31 01:40:31.006870 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:31.006834    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn" event={"ID":"4640ac90-9987-4201-a94e-56a3102f0f0e","Type":"ContainerStarted","Data":"6ca3c61e7cd270d0a2ca8047384d65b23a7837988c323dcb2a770212b5f71024"}
Oct 31 01:40:31.007276 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:31.006894    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn" event={"ID":"4640ac90-9987-4201-a94e-56a3102f0f0e","Type":"ContainerStarted","Data":"90d480a80cd7e0ba7a121c0083487fd91d01fe4bd83f0a79a4ceacfbff37dc07"}
Oct 31 01:40:31.007276 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:31.006924    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn"
Oct 31 01:40:31.008619 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:31.008594    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq" event={"ID":"0731e478-fead-48bf-9de2-4cf3ee6f184c","Type":"ContainerStarted","Data":"b9006d3caeb1330191880231006d93a49bbd94c5e7329a01cfac2aafaaf2e190"}
Oct 31 01:40:31.008729 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:31.008643    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq"
Oct 31 01:40:31.027662 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:31.027603    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn" podStartSLOduration=1.027587397 podStartE2EDuration="1.027587397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:31.026937022 +0000 UTC m=+1187.030651314" watchObservedRunningTime="2026-10-31 01:40:31.027587397 +0000 UTC m=+1187.031301688"
Oct 31 01:40:31.051306 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:31.051244    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq" podStartSLOduration=1.563369184 podStartE2EDuration="2.051228233s" totalImagesPullingTime="487.859049ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:31.049295145 +0000 UTC m=+1187.053009439" watchObservedRunningTime="2026-10-31 01:40:31.051228233 +0000 UTC m=+1187.054942526"
Oct 31 01:40:36.969186 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:36.969154    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw"
Oct 31 01:40:36.994676 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:36.994643    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p"
Oct 31 01:40:37.042607 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:37.042574    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq"
Oct 31 01:40:42.046715 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:42.045682    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn"
Oct 31 01:40:43.007671 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:43.007629    5650 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:40:43.008549 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:43.008492    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8" podUID="af0ee471-3ebf-426f-88e8-689ddc123d8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:40:46.189419 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:46.189380    5650 generic.go:356] "Generic (PLEG): container finished" podID="af0ee471-3ebf-426f-88e8-689ddc123d8e" containerID="fe2949ea3820d9e9e31cc129190a905dc7b124e5b6c3d8528eac8f17423881f1" exitCode=0
Oct 31 01:40:46.189913 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:46.189471    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8" event={"ID":"af0ee471-3ebf-426f-88e8-689ddc123d8e","Type":"ContainerDied","Data":"fe2949ea3820d9e9e31cc129190a905dc7b124e5b6c3d8528eac8f17423881f1"}
Oct 31 01:40:46.189913 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:46.189895    5650 scope.go:122] "RemoveContainer" containerID="fe2949ea3820d9e9e31cc129190a905dc7b124e5b6c3d8528eac8f17423881f1"
Oct 31 01:40:46.191287 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:46.191258    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:40:46.191403 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:46.191344    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:40:46.191403 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:46.191391    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:40:47.198906 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.198873    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8" event={"ID":"af0ee471-3ebf-426f-88e8-689ddc123d8e","Type":"ContainerStarted","Data":"e922da61c00fc4f4fc74f93416db756b022d7165e2f7329716954140e3d6d8a2"}
Oct 31 01:40:47.388649 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.385939    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"]
Oct 31 01:40:47.389604 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.389576    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:40:47.424472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.419395    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"]
Oct 31 01:40:47.424472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.419978    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-0e8b5c\"/\"simulator-w1-0e8b5c-kserve-self-signed-certs\""
Oct 31 01:40:47.424472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.420093    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-0e8b5c\"/\"kube-root-ca.crt\""
Oct 31 01:40:47.424472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.420178    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-0e8b5c\"/\"openshift-service-ca.crt\""
Oct 31 01:40:47.424472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.420273    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-0e8b5c\"/\"default-dockercfg-rspfz\""
Oct 31 01:40:47.540876 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.540828    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-home\") pod \"simulator-w1-0e8b5c-kserve-89647c967-7nbb5\" (UID: \"176ca180-029d-4252-9ce9-a73297ffb286\") " pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:40:47.540876 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.540882    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ql72\" (UniqueName: \"kubernetes.io/projected/176ca180-029d-4252-9ce9-a73297ffb286-kube-api-access-4ql72\") pod \"simulator-w1-0e8b5c-kserve-89647c967-7nbb5\" (UID: \"176ca180-029d-4252-9ce9-a73297ffb286\") " pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:40:47.543055 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.540965    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-tmp-dir\") pod \"simulator-w1-0e8b5c-kserve-89647c967-7nbb5\" (UID: \"176ca180-029d-4252-9ce9-a73297ffb286\") " pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:40:47.543055 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.541042    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-dshm\") pod \"simulator-w1-0e8b5c-kserve-89647c967-7nbb5\" (UID: \"176ca180-029d-4252-9ce9-a73297ffb286\") " pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:40:47.543055 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.541115    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-model-cache\") pod \"simulator-w1-0e8b5c-kserve-89647c967-7nbb5\" (UID: \"176ca180-029d-4252-9ce9-a73297ffb286\") " pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:40:47.543055 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.541193    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176ca180-029d-4252-9ce9-a73297ffb286-tls-certs\") pod \"simulator-w1-0e8b5c-kserve-89647c967-7nbb5\" (UID: \"176ca180-029d-4252-9ce9-a73297ffb286\") " pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:40:47.641956 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.641925    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-home\") pod \"simulator-w1-0e8b5c-kserve-89647c967-7nbb5\" (UID: \"176ca180-029d-4252-9ce9-a73297ffb286\") " pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:40:47.642096 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.641967    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ql72\" (UniqueName: \"kubernetes.io/projected/176ca180-029d-4252-9ce9-a73297ffb286-kube-api-access-4ql72\") pod \"simulator-w1-0e8b5c-kserve-89647c967-7nbb5\" (UID: \"176ca180-029d-4252-9ce9-a73297ffb286\") " pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:40:47.642096 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.641985    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-tmp-dir\") pod \"simulator-w1-0e8b5c-kserve-89647c967-7nbb5\" (UID: \"176ca180-029d-4252-9ce9-a73297ffb286\") " pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:40:47.642096 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.642004    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-dshm\") pod \"simulator-w1-0e8b5c-kserve-89647c967-7nbb5\" (UID: \"176ca180-029d-4252-9ce9-a73297ffb286\") " pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:40:47.642394 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.642360    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-model-cache\") pod \"simulator-w1-0e8b5c-kserve-89647c967-7nbb5\" (UID: \"176ca180-029d-4252-9ce9-a73297ffb286\") " pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:40:47.642465 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.642431    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176ca180-029d-4252-9ce9-a73297ffb286-tls-certs\") pod \"simulator-w1-0e8b5c-kserve-89647c967-7nbb5\" (UID: \"176ca180-029d-4252-9ce9-a73297ffb286\") " pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:40:47.642691 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.642633    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-tmp-dir\") pod \"simulator-w1-0e8b5c-kserve-89647c967-7nbb5\" (UID: \"176ca180-029d-4252-9ce9-a73297ffb286\") " pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:40:47.642691 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.642687    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-model-cache\") pod \"simulator-w1-0e8b5c-kserve-89647c967-7nbb5\" (UID: \"176ca180-029d-4252-9ce9-a73297ffb286\") " pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:40:47.643122 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.643101    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-home\") pod \"simulator-w1-0e8b5c-kserve-89647c967-7nbb5\" (UID: \"176ca180-029d-4252-9ce9-a73297ffb286\") " pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:40:47.645479 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.645083    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-dshm\") pod \"simulator-w1-0e8b5c-kserve-89647c967-7nbb5\" (UID: \"176ca180-029d-4252-9ce9-a73297ffb286\") " pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:40:47.645479 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.645465    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176ca180-029d-4252-9ce9-a73297ffb286-tls-certs\") pod \"simulator-w1-0e8b5c-kserve-89647c967-7nbb5\" (UID: \"176ca180-029d-4252-9ce9-a73297ffb286\") " pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:40:47.656169 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.656139    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ql72\" (UniqueName: \"kubernetes.io/projected/176ca180-029d-4252-9ce9-a73297ffb286-kube-api-access-4ql72\") pod \"simulator-w1-0e8b5c-kserve-89647c967-7nbb5\" (UID: \"176ca180-029d-4252-9ce9-a73297ffb286\") " pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:40:47.683984 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.683946    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:47.684206 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.684182    5650 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 01:40:47.684260 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.684234    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8" podUID="af0ee471-3ebf-426f-88e8-689ddc123d8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 01:40:47.732951 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:47.732868    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:40:48.040345 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:48.040312    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"]
Oct 31 01:40:48.042060 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:40:48.042033    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod176ca180_029d_4252_9ce9_a73297ffb286.slice/crio-e6b24f8c08d2e7d66229c687a28bd5aa0e3f225d6855f222737d08018c08f99a WatchSource:0}: Error finding container e6b24f8c08d2e7d66229c687a28bd5aa0e3f225d6855f222737d08018c08f99a: Status 404 returned error can't find the container with id e6b24f8c08d2e7d66229c687a28bd5aa0e3f225d6855f222737d08018c08f99a
Oct 31 01:40:48.234123 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:48.234080    5650 generic.go:356] "Generic (PLEG): container finished" podID="7f8ad70f-aced-426e-8d7a-d1c17c256582" containerID="b564f738c4a77bb8f71c143af141ec5494444da818473d63c951295aa1db2580" exitCode=0
Oct 31 01:40:48.234546 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:48.234348    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv" event={"ID":"7f8ad70f-aced-426e-8d7a-d1c17c256582","Type":"ContainerDied","Data":"b564f738c4a77bb8f71c143af141ec5494444da818473d63c951295aa1db2580"}
Oct 31 01:40:48.234968 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:48.234949    5650 scope.go:122] "RemoveContainer" containerID="b564f738c4a77bb8f71c143af141ec5494444da818473d63c951295aa1db2580"
Oct 31 01:40:48.236587 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:48.236552    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:40:48.236667 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:48.236639    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:40:48.236702 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:48.236681    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:40:48.239402 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:48.239377    5650 generic.go:356] "Generic (PLEG): container finished" podID="cd6018a4-4e7e-410a-892f-6b3f800addd7" containerID="ef3c2fc9fedc601a7e8676e784a2668eca04022ab412a13713042dde38b7b5e7" exitCode=0
Oct 31 01:40:48.240780 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:48.240759    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj" event={"ID":"cd6018a4-4e7e-410a-892f-6b3f800addd7","Type":"ContainerDied","Data":"ef3c2fc9fedc601a7e8676e784a2668eca04022ab412a13713042dde38b7b5e7"}
Oct 31 01:40:48.241741 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:48.241726    5650 scope.go:122] "RemoveContainer" containerID="ef3c2fc9fedc601a7e8676e784a2668eca04022ab412a13713042dde38b7b5e7"
Oct 31 01:40:48.248512 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:48.248483    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" event={"ID":"176ca180-029d-4252-9ce9-a73297ffb286","Type":"ContainerStarted","Data":"1fe59413b6d15bb32c99320054d153490ebe26888efe9812e3e4162b31b5eeb3"}
Oct 31 01:40:48.248706 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:48.248685    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" event={"ID":"176ca180-029d-4252-9ce9-a73297ffb286","Type":"ContainerStarted","Data":"e6b24f8c08d2e7d66229c687a28bd5aa0e3f225d6855f222737d08018c08f99a"}
Oct 31 01:40:48.251386 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:48.251280    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:40:48.251386 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:48.251351    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:40:48.251386 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:48.251381    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:40:48.295968 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:48.295835    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podStartSLOduration=1.295813833 podStartE2EDuration="1.295813833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:48.291703719 +0000 UTC m=+1204.295418024" watchObservedRunningTime="2026-10-31 01:40:48.295813833 +0000 UTC m=+1204.299528125"
Oct 31 01:40:48.684842 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:48.684743    5650 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 01:40:48.684842 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:48.684808    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8" podUID="af0ee471-3ebf-426f-88e8-689ddc123d8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 01:40:48.733650 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:48.733611    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:40:49.277170 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:49.277129    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv" event={"ID":"7f8ad70f-aced-426e-8d7a-d1c17c256582","Type":"ContainerStarted","Data":"295c7f3229683f3fc00cfd7de2ef7d1dad2e479f9e9bbdbfa89688581af56a44"}
Oct 31 01:40:49.282493 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:49.282190    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj" event={"ID":"cd6018a4-4e7e-410a-892f-6b3f800addd7","Type":"ContainerStarted","Data":"95e8a7b8df9be5a647234ed5aed5b7341d4b12f4cb18b647fdfd5e03349c45e5"}
Oct 31 01:40:49.487564 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:49.487464    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:49.487771 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:49.487750    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 31 01:40:49.487829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:49.487810    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv" podUID="7f8ad70f-aced-426e-8d7a-d1c17c256582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 31 01:40:49.685214 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:49.685123    5650 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 01:40:49.685214 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:49.685185    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8" podUID="af0ee471-3ebf-426f-88e8-689ddc123d8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 01:40:50.172471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:50.171965    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:50.172471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:50.172110    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 31 01:40:50.172471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:50.172153    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj" podUID="cd6018a4-4e7e-410a-892f-6b3f800addd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 31 01:40:50.488567 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:50.488518    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 31 01:40:50.489002 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:50.488585    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv" podUID="7f8ad70f-aced-426e-8d7a-d1c17c256582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 31 01:40:50.684768 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:50.684731    5650 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 01:40:50.684941 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:50.684796    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8" podUID="af0ee471-3ebf-426f-88e8-689ddc123d8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 01:40:51.173474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:51.170932    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 31 01:40:51.173474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:51.170999    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj" podUID="cd6018a4-4e7e-410a-892f-6b3f800addd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 31 01:40:51.488042 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:51.487996    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 31 01:40:51.488355 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:51.488056    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv" podUID="7f8ad70f-aced-426e-8d7a-d1c17c256582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 31 01:40:51.684495 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:51.684266    5650 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 01:40:51.684495 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:51.684329    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8" podUID="af0ee471-3ebf-426f-88e8-689ddc123d8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 01:40:52.171479 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.170624    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 31 01:40:52.171479 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.170703    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj" podUID="cd6018a4-4e7e-410a-892f-6b3f800addd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 31 01:40:52.488021 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.487976    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 31 01:40:52.488202 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.488043    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv" podUID="7f8ad70f-aced-426e-8d7a-d1c17c256582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 31 01:40:52.670648 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.670607    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"]
Oct 31 01:40:52.679170 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.679117    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:40:52.686748 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.686716    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-tmp-dir\") pod \"simulator-w0-3cf6f4-kserve-8479dd7567-zknb9\" (UID: \"99608207-5e9f-4565-ae8c-88738a9a6218\") " pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:40:52.687794 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.687762    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-model-cache\") pod \"simulator-w0-3cf6f4-kserve-8479dd7567-zknb9\" (UID: \"99608207-5e9f-4565-ae8c-88738a9a6218\") " pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:40:52.688021 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.688003    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-home\") pod \"simulator-w0-3cf6f4-kserve-8479dd7567-zknb9\" (UID: \"99608207-5e9f-4565-ae8c-88738a9a6218\") " pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:40:52.688182 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.688167    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-dshm\") pod \"simulator-w0-3cf6f4-kserve-8479dd7567-zknb9\" (UID: \"99608207-5e9f-4565-ae8c-88738a9a6218\") " pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:40:52.688354 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.688337    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w67j\" (UniqueName: \"kubernetes.io/projected/99608207-5e9f-4565-ae8c-88738a9a6218-kube-api-access-2w67j\") pod \"simulator-w0-3cf6f4-kserve-8479dd7567-zknb9\" (UID: \"99608207-5e9f-4565-ae8c-88738a9a6218\") " pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:40:52.688547 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.688531    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99608207-5e9f-4565-ae8c-88738a9a6218-tls-certs\") pod \"simulator-w0-3cf6f4-kserve-8479dd7567-zknb9\" (UID: \"99608207-5e9f-4565-ae8c-88738a9a6218\") " pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:40:52.690290 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.690267    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"]
Oct 31 01:40:52.693253 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.693230    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3cf6f4\"/\"openshift-service-ca.crt\""
Oct 31 01:40:52.693397 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.693381    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3cf6f4\"/\"default-dockercfg-z5c5f\""
Oct 31 01:40:52.693591 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.693502    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3cf6f4\"/\"kube-root-ca.crt\""
Oct 31 01:40:52.693591 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.693339    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3cf6f4\"/\"simulator-w0-3cf6f4-kserve-self-signed-certs\""
Oct 31 01:40:52.761009 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.760914    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:52.766633 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.766601    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:52.769038 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.769014    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 01:40:52.789525 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.789493    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99608207-5e9f-4565-ae8c-88738a9a6218-tls-certs\") pod \"simulator-w0-3cf6f4-kserve-8479dd7567-zknb9\" (UID: \"99608207-5e9f-4565-ae8c-88738a9a6218\") " pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:40:52.789695 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.789589    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-tmp-dir\") pod \"simulator-w0-3cf6f4-kserve-8479dd7567-zknb9\" (UID: \"99608207-5e9f-4565-ae8c-88738a9a6218\") " pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:40:52.789695 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.789650    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-model-cache\") pod \"simulator-w0-3cf6f4-kserve-8479dd7567-zknb9\" (UID: \"99608207-5e9f-4565-ae8c-88738a9a6218\") " pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:40:52.789695 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.789667    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-home\") pod \"simulator-w0-3cf6f4-kserve-8479dd7567-zknb9\" (UID: \"99608207-5e9f-4565-ae8c-88738a9a6218\") " pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:40:52.789695 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.789691    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-dshm\") pod \"simulator-w0-3cf6f4-kserve-8479dd7567-zknb9\" (UID: \"99608207-5e9f-4565-ae8c-88738a9a6218\") " pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:40:52.789829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.789723    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w67j\" (UniqueName: \"kubernetes.io/projected/99608207-5e9f-4565-ae8c-88738a9a6218-kube-api-access-2w67j\") pod \"simulator-w0-3cf6f4-kserve-8479dd7567-zknb9\" (UID: \"99608207-5e9f-4565-ae8c-88738a9a6218\") " pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:40:52.790701 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.790669    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-tmp-dir\") pod \"simulator-w0-3cf6f4-kserve-8479dd7567-zknb9\" (UID: \"99608207-5e9f-4565-ae8c-88738a9a6218\") " pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:40:52.790846 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.790771    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-home\") pod \"simulator-w0-3cf6f4-kserve-8479dd7567-zknb9\" (UID: \"99608207-5e9f-4565-ae8c-88738a9a6218\") " pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:40:52.790947 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.790929    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-model-cache\") pod \"simulator-w0-3cf6f4-kserve-8479dd7567-zknb9\" (UID: \"99608207-5e9f-4565-ae8c-88738a9a6218\") " pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:40:52.792618 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.792593    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99608207-5e9f-4565-ae8c-88738a9a6218-tls-certs\") pod \"simulator-w0-3cf6f4-kserve-8479dd7567-zknb9\" (UID: \"99608207-5e9f-4565-ae8c-88738a9a6218\") " pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:40:52.793414 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.792889    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-dshm\") pod \"simulator-w0-3cf6f4-kserve-8479dd7567-zknb9\" (UID: \"99608207-5e9f-4565-ae8c-88738a9a6218\") " pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:40:52.866779 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:52.866732    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w67j\" (UniqueName: \"kubernetes.io/projected/99608207-5e9f-4565-ae8c-88738a9a6218-kube-api-access-2w67j\") pod \"simulator-w0-3cf6f4-kserve-8479dd7567-zknb9\" (UID: \"99608207-5e9f-4565-ae8c-88738a9a6218\") " pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:40:53.053975 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:53.053865    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:40:53.171476 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:53.171006    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 31 01:40:53.171476 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:53.171068    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj" podUID="cd6018a4-4e7e-410a-892f-6b3f800addd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 31 01:40:53.389743 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:40:53.389705    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99608207_5e9f_4565_ae8c_88738a9a6218.slice/crio-f9f7a08389a418ab94cf6724ac5530f834c833d0ffa7f8b511a9d04bac16836e WatchSource:0}: Error finding container f9f7a08389a418ab94cf6724ac5530f834c833d0ffa7f8b511a9d04bac16836e: Status 404 returned error can't find the container with id f9f7a08389a418ab94cf6724ac5530f834c833d0ffa7f8b511a9d04bac16836e
Oct 31 01:40:53.390377 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:53.390347    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"]
Oct 31 01:40:53.487703 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:53.487666    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 31 01:40:53.487870 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:53.487731    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv" podUID="7f8ad70f-aced-426e-8d7a-d1c17c256582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 31 01:40:54.170646 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:54.170609    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 31 01:40:54.171040 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:54.170667    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj" podUID="cd6018a4-4e7e-410a-892f-6b3f800addd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 31 01:40:54.356853 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:54.356811    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" event={"ID":"99608207-5e9f-4565-ae8c-88738a9a6218","Type":"ContainerStarted","Data":"a7af19eefb39a9079c7e916df5afd8cec2eba54cfcd9a8a18d1e98cdf30494b1"}
Oct 31 01:40:54.357039 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:54.356868    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" event={"ID":"99608207-5e9f-4565-ae8c-88738a9a6218","Type":"ContainerStarted","Data":"f9f7a08389a418ab94cf6724ac5530f834c833d0ffa7f8b511a9d04bac16836e"}
Oct 31 01:40:54.382005 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:54.381950    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podStartSLOduration=2.3819345800000002 podStartE2EDuration="2.38193458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:54.3804421 +0000 UTC m=+1210.384156397" watchObservedRunningTime="2026-10-31 01:40:54.38193458 +0000 UTC m=+1210.385648871"
Oct 31 01:40:54.487779 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:54.487744    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 31 01:40:54.488003 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:54.487797    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv" podUID="7f8ad70f-aced-426e-8d7a-d1c17c256582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 31 01:40:54.936022 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:54.935935    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:55.054871 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.054833    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:40:55.175338 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.175299    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:55.364162 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.364132    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 01:40:55.408233 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.408202    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"]
Oct 31 01:40:55.421706 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.421670    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:40:55.423108 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.423079    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"]
Oct 31 01:40:55.427774 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.427745    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6ae519\"/\"default-dockercfg-wwmcf\""
Oct 31 01:40:55.431352 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.430812    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6ae519\"/\"simulator-w2-6ae519-kserve-self-signed-certs\""
Oct 31 01:40:55.431586 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.431560    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-6ae519\"/\"kube-root-ca.crt\""
Oct 31 01:40:55.431701 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.431655    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-6ae519\"/\"openshift-service-ca.crt\""
Oct 31 01:40:55.493345 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.493306    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:55.495300 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.493804    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:55.495300 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.494672    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 01:40:55.564081 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.564042    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-model-cache\") pod \"simulator-w2-6ae519-kserve-7785bbbc46-tnvd9\" (UID: \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\") " pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:40:55.564281 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.564147    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nscds\" (UniqueName: \"kubernetes.io/projected/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-kube-api-access-nscds\") pod \"simulator-w2-6ae519-kserve-7785bbbc46-tnvd9\" (UID: \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\") " pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:40:55.564281 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.564215    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-tls-certs\") pod \"simulator-w2-6ae519-kserve-7785bbbc46-tnvd9\" (UID: \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\") " pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:40:55.564281 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.564263    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-dshm\") pod \"simulator-w2-6ae519-kserve-7785bbbc46-tnvd9\" (UID: \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\") " pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:40:55.564484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.564380    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-tmp-dir\") pod \"simulator-w2-6ae519-kserve-7785bbbc46-tnvd9\" (UID: \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\") " pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:40:55.564484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.564422    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-home\") pod \"simulator-w2-6ae519-kserve-7785bbbc46-tnvd9\" (UID: \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\") " pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:40:55.666049 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.665943    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nscds\" (UniqueName: \"kubernetes.io/projected/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-kube-api-access-nscds\") pod \"simulator-w2-6ae519-kserve-7785bbbc46-tnvd9\" (UID: \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\") " pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:40:55.666231 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.666108    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-tls-certs\") pod \"simulator-w2-6ae519-kserve-7785bbbc46-tnvd9\" (UID: \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\") " pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:40:55.666231 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.666166    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-dshm\") pod \"simulator-w2-6ae519-kserve-7785bbbc46-tnvd9\" (UID: \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\") " pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:40:55.666356 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.666260    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-tmp-dir\") pod \"simulator-w2-6ae519-kserve-7785bbbc46-tnvd9\" (UID: \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\") " pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:40:55.666356 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.666307    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-home\") pod \"simulator-w2-6ae519-kserve-7785bbbc46-tnvd9\" (UID: \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\") " pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:40:55.666486 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.666364    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-model-cache\") pod \"simulator-w2-6ae519-kserve-7785bbbc46-tnvd9\" (UID: \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\") " pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:40:55.667114 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.667090    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-model-cache\") pod \"simulator-w2-6ae519-kserve-7785bbbc46-tnvd9\" (UID: \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\") " pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:40:55.667399 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.667154    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-tmp-dir\") pod \"simulator-w2-6ae519-kserve-7785bbbc46-tnvd9\" (UID: \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\") " pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:40:55.667399 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.667337    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-home\") pod \"simulator-w2-6ae519-kserve-7785bbbc46-tnvd9\" (UID: \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\") " pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:40:55.671483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.669084    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-dshm\") pod \"simulator-w2-6ae519-kserve-7785bbbc46-tnvd9\" (UID: \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\") " pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:40:55.671483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.669154    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-tls-certs\") pod \"simulator-w2-6ae519-kserve-7785bbbc46-tnvd9\" (UID: \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\") " pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:40:55.682007 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.681957    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nscds\" (UniqueName: \"kubernetes.io/projected/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-kube-api-access-nscds\") pod \"simulator-w2-6ae519-kserve-7785bbbc46-tnvd9\" (UID: \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\") " pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:40:55.736636 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:55.736594    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:40:56.257173 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:56.257054    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"]
Oct 31 01:40:56.377329 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:56.377289    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" event={"ID":"0cde8fd2-23bc-46c4-8f56-f6e17339c50d","Type":"ContainerStarted","Data":"ba0612ecac6f18630cd148e4628193307b34484517de075c6762cc31d32a0158"}
Oct 31 01:40:57.383748 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:57.383697    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" event={"ID":"0cde8fd2-23bc-46c4-8f56-f6e17339c50d","Type":"ContainerStarted","Data":"be9a7c7decc2311a3c5ce38a880ecc4dcab782986a1adc40ef55fe23d778b931"}
Oct 31 01:40:57.405785 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:57.405726    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podStartSLOduration=2.405712112 podStartE2EDuration="2.405712112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:57.40551949 +0000 UTC m=+1213.409233778" watchObservedRunningTime="2026-10-31 01:40:57.405712112 +0000 UTC m=+1213.409426403"
Oct 31 01:40:57.733303 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:57.733267    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:40:57.737081 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:57.737050    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:40:57.746406 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:57.746379    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:40:58.401252 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:40:58.401218    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:40:58.551578 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:40:58.551545    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:41:03.054813 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:03.054770    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:41:03.067819 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:03.067780    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:41:03.432732 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:03.432647    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:41:05.737528 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:05.737488    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:41:05.751010 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:05.750978    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:41:06.457567 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:06.457534    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:41:12.052472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.052408    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"]
Oct 31 01:41:12.056377 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.056349    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:41:12.066486 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.066233    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-7e5955\"/\"kube-root-ca.crt\""
Oct 31 01:41:12.067572 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.067544    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-7e5955\"/\"openshift-service-ca.crt\""
Oct 31 01:41:12.068485 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.068437    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7e5955\"/\"default-dockercfg-2xhns\""
Oct 31 01:41:12.068566 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.068553    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7e5955\"/\"premium-w3-7e5955-kserve-self-signed-certs\""
Oct 31 01:41:12.069079 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.069062    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"]
Oct 31 01:41:12.189054 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.189014    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-model-cache\") pod \"premium-w3-7e5955-kserve-55f747475-rzgth\" (UID: \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\") " pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:41:12.189276 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.189073    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-tmp-dir\") pod \"premium-w3-7e5955-kserve-55f747475-rzgth\" (UID: \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\") " pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:41:12.189276 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.189162    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8whjd\" (UniqueName: \"kubernetes.io/projected/a99af64c-978c-4d8c-9d5b-260fc08faf6d-kube-api-access-8whjd\") pod \"premium-w3-7e5955-kserve-55f747475-rzgth\" (UID: \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\") " pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:41:12.189276 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.189221    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-home\") pod \"premium-w3-7e5955-kserve-55f747475-rzgth\" (UID: \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\") " pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:41:12.189276 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.189256    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a99af64c-978c-4d8c-9d5b-260fc08faf6d-tls-certs\") pod \"premium-w3-7e5955-kserve-55f747475-rzgth\" (UID: \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\") " pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:41:12.189525 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.189286    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-dshm\") pod \"premium-w3-7e5955-kserve-55f747475-rzgth\" (UID: \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\") " pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:41:12.290532 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.290477    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8whjd\" (UniqueName: \"kubernetes.io/projected/a99af64c-978c-4d8c-9d5b-260fc08faf6d-kube-api-access-8whjd\") pod \"premium-w3-7e5955-kserve-55f747475-rzgth\" (UID: \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\") " pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:41:12.290532 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.290540    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-home\") pod \"premium-w3-7e5955-kserve-55f747475-rzgth\" (UID: \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\") " pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:41:12.290807 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.290564    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a99af64c-978c-4d8c-9d5b-260fc08faf6d-tls-certs\") pod \"premium-w3-7e5955-kserve-55f747475-rzgth\" (UID: \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\") " pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:41:12.290807 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.290582    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-dshm\") pod \"premium-w3-7e5955-kserve-55f747475-rzgth\" (UID: \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\") " pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:41:12.290807 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.290689    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-model-cache\") pod \"premium-w3-7e5955-kserve-55f747475-rzgth\" (UID: \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\") " pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:41:12.290807 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.290711    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-tmp-dir\") pod \"premium-w3-7e5955-kserve-55f747475-rzgth\" (UID: \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\") " pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:41:12.291069 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.291045    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-home\") pod \"premium-w3-7e5955-kserve-55f747475-rzgth\" (UID: \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\") " pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:41:12.291131 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.291086    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-tmp-dir\") pod \"premium-w3-7e5955-kserve-55f747475-rzgth\" (UID: \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\") " pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:41:12.291171 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.291145    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-model-cache\") pod \"premium-w3-7e5955-kserve-55f747475-rzgth\" (UID: \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\") " pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:41:12.292983 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.292926    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-dshm\") pod \"premium-w3-7e5955-kserve-55f747475-rzgth\" (UID: \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\") " pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:41:12.293203 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.293188    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a99af64c-978c-4d8c-9d5b-260fc08faf6d-tls-certs\") pod \"premium-w3-7e5955-kserve-55f747475-rzgth\" (UID: \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\") " pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:41:12.303746 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.303681    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8whjd\" (UniqueName: \"kubernetes.io/projected/a99af64c-978c-4d8c-9d5b-260fc08faf6d-kube-api-access-8whjd\") pod \"premium-w3-7e5955-kserve-55f747475-rzgth\" (UID: \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\") " pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:41:12.370650 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.370609    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:41:12.649691 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:12.649665    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"]
Oct 31 01:41:12.650351 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:41:12.650325    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda99af64c_978c_4d8c_9d5b_260fc08faf6d.slice/crio-cca8747a34fa20f08cafc5b7f873ef7befff4b5caa7608a7df6273a8ad0bff5f WatchSource:0}: Error finding container cca8747a34fa20f08cafc5b7f873ef7befff4b5caa7608a7df6273a8ad0bff5f: Status 404 returned error can't find the container with id cca8747a34fa20f08cafc5b7f873ef7befff4b5caa7608a7df6273a8ad0bff5f
Oct 31 01:41:13.489635 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:13.489599    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" event={"ID":"a99af64c-978c-4d8c-9d5b-260fc08faf6d","Type":"ContainerStarted","Data":"884dffa5e0dd7aa2484d580f5630e271197af7425642bbd44cd33097df4b95b6"}
Oct 31 01:41:13.489635 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:13.489643    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" event={"ID":"a99af64c-978c-4d8c-9d5b-260fc08faf6d","Type":"ContainerStarted","Data":"cca8747a34fa20f08cafc5b7f873ef7befff4b5caa7608a7df6273a8ad0bff5f"}
Oct 31 01:41:14.371224 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:14.371187    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:41:19.447128 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.447062    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podStartSLOduration=7.447045031 podStartE2EDuration="7.447045031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:41:13.509950208 +0000 UTC m=+1229.513664500" watchObservedRunningTime="2026-10-31 01:41:19.447045031 +0000 UTC m=+1235.450759366"
Oct 31 01:41:19.447989 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.447968    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"]
Oct 31 01:41:19.452001 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.451972    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.464238 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.464204    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-be6935-openshift-default-dockercfg-vj9xz\""
Oct 31 01:41:19.469208 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.469171    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"]
Oct 31 01:41:19.564761 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.564715    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.564946 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.564793    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jhvq\" (UniqueName: \"kubernetes.io/projected/94727d94-112b-437f-84f5-bffde7e46284-kube-api-access-9jhvq\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.564946 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.564844    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.564946 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.564937    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94727d94-112b-437f-84f5-bffde7e46284-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.565054 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.564964    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94727d94-112b-437f-84f5-bffde7e46284-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.565054 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.564987    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.565054 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.565027    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.565156 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.565132    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.565187 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.565162    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94727d94-112b-437f-84f5-bffde7e46284-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.666085 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.666048    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jhvq\" (UniqueName: \"kubernetes.io/projected/94727d94-112b-437f-84f5-bffde7e46284-kube-api-access-9jhvq\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.666085 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.666091    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.666352 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.666137    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94727d94-112b-437f-84f5-bffde7e46284-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.666352 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.666159    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94727d94-112b-437f-84f5-bffde7e46284-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.666352 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.666261    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.666352 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.666327    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.666603 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.666360    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.666603 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.666385    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94727d94-112b-437f-84f5-bffde7e46284-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.666603 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.666496    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.666759 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.666637    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.666888 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.666869    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.666965 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.666897    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.667000 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.666982    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.667000 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.666994    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94727d94-112b-437f-84f5-bffde7e46284-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.668932 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.668902    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.669077 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.669058    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94727d94-112b-437f-84f5-bffde7e46284-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.679370 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.679336    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94727d94-112b-437f-84f5-bffde7e46284-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.679539 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.679521    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jhvq\" (UniqueName: \"kubernetes.io/projected/94727d94-112b-437f-84f5-bffde7e46284-kube-api-access-9jhvq\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:19.778681 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:19.778639    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:20.062418 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:20.062390    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"]
Oct 31 01:41:20.064084 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:41:20.064055    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94727d94_112b_437f_84f5_bffde7e46284.slice/crio-da3f6641da6d1faccbf5f049a0a9ce378167e1a3aeb299a3c6a8c86abc632835 WatchSource:0}: Error finding container da3f6641da6d1faccbf5f049a0a9ce378167e1a3aeb299a3c6a8c86abc632835: Status 404 returned error can't find the container with id da3f6641da6d1faccbf5f049a0a9ce378167e1a3aeb299a3c6a8c86abc632835
Oct 31 01:41:20.066665 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:20.066632    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:41:20.066763 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:20.066694    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:41:20.066763 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:20.066724    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:41:20.536304 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:20.536261    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr" event={"ID":"94727d94-112b-437f-84f5-bffde7e46284","Type":"ContainerStarted","Data":"43d5f1809c05ccb9a4e29c496e5947caabc61b78643e9ff8b82a085a631411ab"}
Oct 31 01:41:20.536748 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:20.536323    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr" event={"ID":"94727d94-112b-437f-84f5-bffde7e46284","Type":"ContainerStarted","Data":"da3f6641da6d1faccbf5f049a0a9ce378167e1a3aeb299a3c6a8c86abc632835"}
Oct 31 01:41:20.566088 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:20.566025    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr" podStartSLOduration=1.566010262 podStartE2EDuration="1.566010262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:41:20.562390361 +0000 UTC m=+1236.566104653" watchObservedRunningTime="2026-10-31 01:41:20.566010262 +0000 UTC m=+1236.569724554"
Oct 31 01:41:20.779324 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:20.779280    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:21.781509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:21.781473    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 31 01:41:21.781914 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:21.781526    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr" podUID="94727d94-112b-437f-84f5-bffde7e46284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 31 01:41:22.371711 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:22.371667    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:41:22.384238 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:22.384205    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:41:22.561704 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:22.561671    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:41:22.779900 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:22.779863    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 31 01:41:22.780088 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:22.779918    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr" podUID="94727d94-112b-437f-84f5-bffde7e46284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 31 01:41:23.779936 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:23.779894    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 31 01:41:23.780345 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:23.779956    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr" podUID="94727d94-112b-437f-84f5-bffde7e46284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 31 01:41:24.779417 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:24.779379    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 31 01:41:24.779616 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:24.779433    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr" podUID="94727d94-112b-437f-84f5-bffde7e46284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 31 01:41:25.779425 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:25.779382    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 31 01:41:25.779842 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:25.779474    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr" podUID="94727d94-112b-437f-84f5-bffde7e46284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 31 01:41:26.900715 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:26.900673    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:26.901543 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:26.901513    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:26.902574 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:26.902553    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 01:41:28.555199 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:41:28.555155    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:41:54.345369 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.345330    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"]
Oct 31 01:41:54.350406 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.350385    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.360673 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.360631    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-0e8b5c-openshift-default-dockercfg-pq7tc\""
Oct 31 01:41:54.363353 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.363321    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"]
Oct 31 01:41:54.413251 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.413205    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.413251 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.413255    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.413493 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.413373    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.413493 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.413411    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.413569 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.413505    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.413569 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.413527    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.413569 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.413551    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.413711 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.413686    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mxjj\" (UniqueName: \"kubernetes.io/projected/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-kube-api-access-5mxjj\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.413749 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.413735    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.514688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.514643    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.514688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.514693    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.514991 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.514734    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mxjj\" (UniqueName: \"kubernetes.io/projected/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-kube-api-access-5mxjj\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.514991 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.514763    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.514991 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.514820    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.514991 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.514847    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.514991 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.514895    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.514991 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.514931    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.514991 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.514990    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.515357 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.515315    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.515413 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.515387    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.515504 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.515482    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.515574 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.515488    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.515738 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.515716    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.517224 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.517200    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.517552 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.517530    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.528116 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.528090    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.528237 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.528156    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mxjj\" (UniqueName: \"kubernetes.io/projected/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-kube-api-access-5mxjj\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.666009 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.665914    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:54.954961 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.954931    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"]
Oct 31 01:41:54.956419 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:41:54.956393    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3603d0_f3c8_4bbc_96c7_366b59bffdce.slice/crio-1d4953b8738e0ba821cd263ab36e9bf28d9a9ba8a174c4f9fa8107bdb654018f WatchSource:0}: Error finding container 1d4953b8738e0ba821cd263ab36e9bf28d9a9ba8a174c4f9fa8107bdb654018f: Status 404 returned error can't find the container with id 1d4953b8738e0ba821cd263ab36e9bf28d9a9ba8a174c4f9fa8107bdb654018f
Oct 31 01:41:54.958747 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.958712    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:41:54.958841 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.958786    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:41:54.958841 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:54.958816    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:41:55.777879 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:55.777829    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc" event={"ID":"bb3603d0-f3c8-4bbc-96c7-366b59bffdce","Type":"ContainerStarted","Data":"1b460b3b1717dfb6801690ad4515a7f922a63a29cc4c0260e4f3bd715062a724"}
Oct 31 01:41:55.778431 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:55.777901    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc" event={"ID":"bb3603d0-f3c8-4bbc-96c7-366b59bffdce","Type":"ContainerStarted","Data":"1d4953b8738e0ba821cd263ab36e9bf28d9a9ba8a174c4f9fa8107bdb654018f"}
Oct 31 01:41:55.824296 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:55.824223    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc" podStartSLOduration=1.8242009879999999 podStartE2EDuration="1.824200988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:41:55.820950763 +0000 UTC m=+1271.824665054" watchObservedRunningTime="2026-10-31 01:41:55.824200988 +0000 UTC m=+1271.827915281"
Oct 31 01:41:56.666530 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:56.666480    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:41:56.668126 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:56.668102    5650 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 01:41:56.668185 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:56.668166    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc" podUID="bb3603d0-f3c8-4bbc-96c7-366b59bffdce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 01:41:57.666340 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:57.666300    5650 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 01:41:57.666867 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:57.666380    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc" podUID="bb3603d0-f3c8-4bbc-96c7-366b59bffdce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 01:41:58.568579 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:41:58.568534    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:41:58.669949 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:58.669911    5650 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 01:41:58.670546 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:58.670510    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc" podUID="bb3603d0-f3c8-4bbc-96c7-366b59bffdce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 01:41:59.666947 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:59.666907    5650 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 01:41:59.667168 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:41:59.666988    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc" podUID="bb3603d0-f3c8-4bbc-96c7-366b59bffdce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 01:42:00.228179 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:00.228130    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-767349a2-ph52r"]
Oct 31 01:42:00.235735 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:00.235702    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-767349a2-ph52r"]
Oct 31 01:42:00.235964 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:00.235810    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767349a2-ph52r"
Oct 31 01:42:00.241127 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:00.241090    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:42:00.241362 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:00.241167    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:42:00.262432 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:00.262392    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40fb5d82-5d28-4c48-acab-f5a30ea6edce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-767349a2-ph52r\" (UID: \"40fb5d82-5d28-4c48-acab-f5a30ea6edce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767349a2-ph52r"
Oct 31 01:42:00.262612 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:00.262592    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frsjm\" (UniqueName: \"kubernetes.io/projected/40fb5d82-5d28-4c48-acab-f5a30ea6edce-kube-api-access-frsjm\") pod \"maas-api-revoke-sub-767349a2-ph52r\" (UID: \"40fb5d82-5d28-4c48-acab-f5a30ea6edce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767349a2-ph52r"
Oct 31 01:42:00.262677 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:00.262632    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40fb5d82-5d28-4c48-acab-f5a30ea6edce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-767349a2-ph52r\" (UID: \"40fb5d82-5d28-4c48-acab-f5a30ea6edce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767349a2-ph52r"
Oct 31 01:42:00.363528 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:00.363487    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40fb5d82-5d28-4c48-acab-f5a30ea6edce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-767349a2-ph52r\" (UID: \"40fb5d82-5d28-4c48-acab-f5a30ea6edce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767349a2-ph52r"
Oct 31 01:42:00.363735 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:00.363615    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frsjm\" (UniqueName: \"kubernetes.io/projected/40fb5d82-5d28-4c48-acab-f5a30ea6edce-kube-api-access-frsjm\") pod \"maas-api-revoke-sub-767349a2-ph52r\" (UID: \"40fb5d82-5d28-4c48-acab-f5a30ea6edce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767349a2-ph52r"
Oct 31 01:42:00.363735 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:00.363637    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40fb5d82-5d28-4c48-acab-f5a30ea6edce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-767349a2-ph52r\" (UID: \"40fb5d82-5d28-4c48-acab-f5a30ea6edce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767349a2-ph52r"
Oct 31 01:42:00.364515 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:00.364492    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40fb5d82-5d28-4c48-acab-f5a30ea6edce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-767349a2-ph52r\" (UID: \"40fb5d82-5d28-4c48-acab-f5a30ea6edce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767349a2-ph52r"
Oct 31 01:42:00.366968 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:00.366943    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40fb5d82-5d28-4c48-acab-f5a30ea6edce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-767349a2-ph52r\" (UID: \"40fb5d82-5d28-4c48-acab-f5a30ea6edce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767349a2-ph52r"
Oct 31 01:42:00.377794 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:00.377767    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frsjm\" (UniqueName: \"kubernetes.io/projected/40fb5d82-5d28-4c48-acab-f5a30ea6edce-kube-api-access-frsjm\") pod \"maas-api-revoke-sub-767349a2-ph52r\" (UID: \"40fb5d82-5d28-4c48-acab-f5a30ea6edce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767349a2-ph52r"
Oct 31 01:42:00.552232 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:00.552145    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767349a2-ph52r"
Oct 31 01:42:00.666868 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:00.666817    5650 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 01:42:00.667057 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:00.666891    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc" podUID="bb3603d0-f3c8-4bbc-96c7-366b59bffdce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 01:42:00.859049 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:00.859024    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-767349a2-ph52r"]
Oct 31 01:42:00.860307 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:42:00.860272    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40fb5d82_5d28_4c48_acab_f5a30ea6edce.slice/crio-06c6c4ee23ab5b296570f99b8384addab51fa6043e0960a35cfaa5dd634d74eb WatchSource:0}: Error finding container 06c6c4ee23ab5b296570f99b8384addab51fa6043e0960a35cfaa5dd634d74eb: Status 404 returned error can't find the container with id 06c6c4ee23ab5b296570f99b8384addab51fa6043e0960a35cfaa5dd634d74eb
Oct 31 01:42:01.760574 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:01.760532    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:42:01.761504 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:01.761477    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:42:01.762413 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:01.762393    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 01:42:01.860607 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:01.860572    5650 generic.go:356] "Generic (PLEG): container finished" podID="40fb5d82-5d28-4c48-acab-f5a30ea6edce" containerID="85f513b2a91f54dc1286917b41aa6951f7e2a48fa1790eacc63a9cccdb5ad8c6" exitCode=0
Oct 31 01:42:01.860796 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:01.860655    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767349a2-ph52r" event={"ID":"40fb5d82-5d28-4c48-acab-f5a30ea6edce","Type":"ContainerDied","Data":"85f513b2a91f54dc1286917b41aa6951f7e2a48fa1790eacc63a9cccdb5ad8c6"}
Oct 31 01:42:01.860796 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:01.860714    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767349a2-ph52r" event={"ID":"40fb5d82-5d28-4c48-acab-f5a30ea6edce","Type":"ContainerStarted","Data":"06c6c4ee23ab5b296570f99b8384addab51fa6043e0960a35cfaa5dd634d74eb"}
Oct 31 01:42:03.025819 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:03.025790    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767349a2-ph52r"
Oct 31 01:42:03.095984 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:03.095945    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40fb5d82-5d28-4c48-acab-f5a30ea6edce-kube-api-access-frsjm\" (UniqueName: \"kubernetes.io/projected/40fb5d82-5d28-4c48-acab-f5a30ea6edce-kube-api-access-frsjm\") pod \"40fb5d82-5d28-4c48-acab-f5a30ea6edce\" (UID: \"40fb5d82-5d28-4c48-acab-f5a30ea6edce\") "
Oct 31 01:42:03.096190 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:03.096038    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40fb5d82-5d28-4c48-acab-f5a30ea6edce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40fb5d82-5d28-4c48-acab-f5a30ea6edce-maas-api-xks-ca\") pod \"40fb5d82-5d28-4c48-acab-f5a30ea6edce\" (UID: \"40fb5d82-5d28-4c48-acab-f5a30ea6edce\") "
Oct 31 01:42:03.096190 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:03.096072    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40fb5d82-5d28-4c48-acab-f5a30ea6edce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40fb5d82-5d28-4c48-acab-f5a30ea6edce-maas-api-openshift-ca\") pod \"40fb5d82-5d28-4c48-acab-f5a30ea6edce\" (UID: \"40fb5d82-5d28-4c48-acab-f5a30ea6edce\") "
Oct 31 01:42:03.096590 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:03.096558    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40fb5d82-5d28-4c48-acab-f5a30ea6edce-maas-api-openshift-ca" pod "40fb5d82-5d28-4c48-acab-f5a30ea6edce" (UID: "40fb5d82-5d28-4c48-acab-f5a30ea6edce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:42:03.098348 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:03.098323    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40fb5d82-5d28-4c48-acab-f5a30ea6edce-kube-api-access-frsjm" pod "40fb5d82-5d28-4c48-acab-f5a30ea6edce" (UID: "40fb5d82-5d28-4c48-acab-f5a30ea6edce"). InnerVolumeSpecName "kube-api-access-frsjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:03.098511 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:03.098364    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40fb5d82-5d28-4c48-acab-f5a30ea6edce-maas-api-xks-ca" pod "40fb5d82-5d28-4c48-acab-f5a30ea6edce" (UID: "40fb5d82-5d28-4c48-acab-f5a30ea6edce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:42:03.198351 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:03.198248    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frsjm\" (UniqueName: \"kubernetes.io/projected/40fb5d82-5d28-4c48-acab-f5a30ea6edce-kube-api-access-frsjm\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:03.198351 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:03.198292    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40fb5d82-5d28-4c48-acab-f5a30ea6edce-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:03.198351 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:03.198308    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40fb5d82-5d28-4c48-acab-f5a30ea6edce-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:03.872803 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:03.872765    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767349a2-ph52r"
Oct 31 01:42:03.872983 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:03.872803    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767349a2-ph52r" event={"ID":"40fb5d82-5d28-4c48-acab-f5a30ea6edce","Type":"ContainerDied","Data":"06c6c4ee23ab5b296570f99b8384addab51fa6043e0960a35cfaa5dd634d74eb"}
Oct 31 01:42:03.872983 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:03.872875    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06c6c4ee23ab5b296570f99b8384addab51fa6043e0960a35cfaa5dd634d74eb"
Oct 31 01:42:13.469956 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:13.469912    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-84dc6d9888-zklck"]
Oct 31 01:42:13.471638 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:13.471595    5650 cpu_manager.go:414] "containerMap: removing container" podUID="40fb5d82-5d28-4c48-acab-f5a30ea6edce" containerName="revoke-keys"
Oct 31 01:42:13.471785 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:13.471770    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40fb5d82-5d28-4c48-acab-f5a30ea6edce" containerName="revoke-keys"
Oct 31 01:42:13.472076 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:13.472058    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="40fb5d82-5d28-4c48-acab-f5a30ea6edce" containerName="revoke-keys"
Oct 31 01:42:13.476576 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:13.476546    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84dc6d9888-zklck"
Oct 31 01:42:13.482221 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:13.482183    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 01:42:13.482363 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:13.482241    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 01:42:13.482363 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:13.482265    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 01:42:13.482363 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:13.482319    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 01:42:13.482709 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:13.482685    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-84dc6d9888-zklck"]
Oct 31 01:42:13.483274 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:13.483253    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-e1bb458aa6fe\""
Oct 31 01:42:13.483402 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:13.483326    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b6fc1f12130d\""
Oct 31 01:42:13.483662 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:13.483632    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-h627d\""
Oct 31 01:42:13.680978 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:13.680935    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-plugins-config-volume\") pod \"payload-processing-external-model-84dc6d9888-zklck\" (UID: \"dc6371d3-66fb-4c1e-86a8-ddfd27197a14\") " pod="models-as-a-service/payload-processing-external-model-84dc6d9888-zklck"
Oct 31 01:42:13.681178 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:13.681015    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-routing-overlay-volume\") pod \"payload-processing-external-model-84dc6d9888-zklck\" (UID: \"dc6371d3-66fb-4c1e-86a8-ddfd27197a14\") " pod="models-as-a-service/payload-processing-external-model-84dc6d9888-zklck"
Oct 31 01:42:13.681242 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:13.681221    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jd9z\" (UniqueName: \"kubernetes.io/projected/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-kube-api-access-9jd9z\") pod \"payload-processing-external-model-84dc6d9888-zklck\" (UID: \"dc6371d3-66fb-4c1e-86a8-ddfd27197a14\") " pod="models-as-a-service/payload-processing-external-model-84dc6d9888-zklck"
Oct 31 01:42:13.681373 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:13.681346    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-provider-credentials\") pod \"payload-processing-external-model-84dc6d9888-zklck\" (UID: \"dc6371d3-66fb-4c1e-86a8-ddfd27197a14\") " pod="models-as-a-service/payload-processing-external-model-84dc6d9888-zklck"
Oct 31 01:42:13.782865 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:13.782764    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-provider-credentials\") pod \"payload-processing-external-model-84dc6d9888-zklck\" (UID: \"dc6371d3-66fb-4c1e-86a8-ddfd27197a14\") " pod="models-as-a-service/payload-processing-external-model-84dc6d9888-zklck"
Oct 31 01:42:13.783028 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:13.782924    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-plugins-config-volume\") pod \"payload-processing-external-model-84dc6d9888-zklck\" (UID: \"dc6371d3-66fb-4c1e-86a8-ddfd27197a14\") " pod="models-as-a-service/payload-processing-external-model-84dc6d9888-zklck"
Oct 31 01:42:13.783028 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:13.782977    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-routing-overlay-volume\") pod \"payload-processing-external-model-84dc6d9888-zklck\" (UID: \"dc6371d3-66fb-4c1e-86a8-ddfd27197a14\") " pod="models-as-a-service/payload-processing-external-model-84dc6d9888-zklck"
Oct 31 01:42:13.783145 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:13.783071    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jd9z\" (UniqueName: \"kubernetes.io/projected/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-kube-api-access-9jd9z\") pod \"payload-processing-external-model-84dc6d9888-zklck\" (UID: \"dc6371d3-66fb-4c1e-86a8-ddfd27197a14\") " pod="models-as-a-service/payload-processing-external-model-84dc6d9888-zklck"
Oct 31 01:42:13.783841 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:13.783816    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-plugins-config-volume\") pod \"payload-processing-external-model-84dc6d9888-zklck\" (UID: \"dc6371d3-66fb-4c1e-86a8-ddfd27197a14\") " pod="models-as-a-service/payload-processing-external-model-84dc6d9888-zklck"
Oct 31 01:42:13.783965 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:13.783846    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-routing-overlay-volume\") pod \"payload-processing-external-model-84dc6d9888-zklck\" (UID: \"dc6371d3-66fb-4c1e-86a8-ddfd27197a14\") " pod="models-as-a-service/payload-processing-external-model-84dc6d9888-zklck"
Oct 31 01:42:13.785623 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:13.785599    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-provider-credentials\") pod \"payload-processing-external-model-84dc6d9888-zklck\" (UID: \"dc6371d3-66fb-4c1e-86a8-ddfd27197a14\") " pod="models-as-a-service/payload-processing-external-model-84dc6d9888-zklck"
Oct 31 01:42:13.803597 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:13.803561    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jd9z\" (UniqueName: \"kubernetes.io/projected/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-kube-api-access-9jd9z\") pod \"payload-processing-external-model-84dc6d9888-zklck\" (UID: \"dc6371d3-66fb-4c1e-86a8-ddfd27197a14\") " pod="models-as-a-service/payload-processing-external-model-84dc6d9888-zklck"
Oct 31 01:42:14.018874 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.018828    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv"]
Oct 31 01:42:14.022995 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.022970    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv"
Oct 31 01:42:14.037948 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.034819    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-735cc15d15ca\""
Oct 31 01:42:14.037948 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.034877    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-efc045041ce5\""
Oct 31 01:42:14.037948 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.036178    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv"]
Oct 31 01:42:14.086461 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.086398    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad91b85-a022-47b9-9872-4601a6b3215b-plugins-config-volume\") pod \"payload-processing-external-model-69d6d4dd5c-xrldv\" (UID: \"4ad91b85-a022-47b9-9872-4601a6b3215b\") " pod="models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv"
Oct 31 01:42:14.086669 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.086527    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ad91b85-a022-47b9-9872-4601a6b3215b-provider-credentials\") pod \"payload-processing-external-model-69d6d4dd5c-xrldv\" (UID: \"4ad91b85-a022-47b9-9872-4601a6b3215b\") " pod="models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv"
Oct 31 01:42:14.086669 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.086626    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ad91b85-a022-47b9-9872-4601a6b3215b-routing-overlay-volume\") pod \"payload-processing-external-model-69d6d4dd5c-xrldv\" (UID: \"4ad91b85-a022-47b9-9872-4601a6b3215b\") " pod="models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv"
Oct 31 01:42:14.086801 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.086712    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5sp5\" (UniqueName: \"kubernetes.io/projected/4ad91b85-a022-47b9-9872-4601a6b3215b-kube-api-access-b5sp5\") pod \"payload-processing-external-model-69d6d4dd5c-xrldv\" (UID: \"4ad91b85-a022-47b9-9872-4601a6b3215b\") " pod="models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv"
Oct 31 01:42:14.097319 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.097280    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84dc6d9888-zklck"
Oct 31 01:42:14.194386 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.191822    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ad91b85-a022-47b9-9872-4601a6b3215b-routing-overlay-volume\") pod \"payload-processing-external-model-69d6d4dd5c-xrldv\" (UID: \"4ad91b85-a022-47b9-9872-4601a6b3215b\") " pod="models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv"
Oct 31 01:42:14.194386 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.191906    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5sp5\" (UniqueName: \"kubernetes.io/projected/4ad91b85-a022-47b9-9872-4601a6b3215b-kube-api-access-b5sp5\") pod \"payload-processing-external-model-69d6d4dd5c-xrldv\" (UID: \"4ad91b85-a022-47b9-9872-4601a6b3215b\") " pod="models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv"
Oct 31 01:42:14.194386 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.191981    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad91b85-a022-47b9-9872-4601a6b3215b-plugins-config-volume\") pod \"payload-processing-external-model-69d6d4dd5c-xrldv\" (UID: \"4ad91b85-a022-47b9-9872-4601a6b3215b\") " pod="models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv"
Oct 31 01:42:14.194386 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.192049    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ad91b85-a022-47b9-9872-4601a6b3215b-provider-credentials\") pod \"payload-processing-external-model-69d6d4dd5c-xrldv\" (UID: \"4ad91b85-a022-47b9-9872-4601a6b3215b\") " pod="models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv"
Oct 31 01:42:14.194386 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.193786    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ad91b85-a022-47b9-9872-4601a6b3215b-routing-overlay-volume\") pod \"payload-processing-external-model-69d6d4dd5c-xrldv\" (UID: \"4ad91b85-a022-47b9-9872-4601a6b3215b\") " pod="models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv"
Oct 31 01:42:14.194953 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.194900    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad91b85-a022-47b9-9872-4601a6b3215b-plugins-config-volume\") pod \"payload-processing-external-model-69d6d4dd5c-xrldv\" (UID: \"4ad91b85-a022-47b9-9872-4601a6b3215b\") " pod="models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv"
Oct 31 01:42:14.195437 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.195412    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ad91b85-a022-47b9-9872-4601a6b3215b-provider-credentials\") pod \"payload-processing-external-model-69d6d4dd5c-xrldv\" (UID: \"4ad91b85-a022-47b9-9872-4601a6b3215b\") " pod="models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv"
Oct 31 01:42:14.218496 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.214227    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5sp5\" (UniqueName: \"kubernetes.io/projected/4ad91b85-a022-47b9-9872-4601a6b3215b-kube-api-access-b5sp5\") pod \"payload-processing-external-model-69d6d4dd5c-xrldv\" (UID: \"4ad91b85-a022-47b9-9872-4601a6b3215b\") " pod="models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv"
Oct 31 01:42:14.342702 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.342600    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv"
Oct 31 01:42:14.395631 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.395596    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-84dc6d9888-zklck"]
Oct 31 01:42:14.627654 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.627619    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv"]
Oct 31 01:42:14.631527 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:42:14.631489    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad91b85_a022_47b9_9872_4601a6b3215b.slice/crio-45aabeb4645b15599906063eb5eb9b41a2f90947ea792911fac81b0985b3afbd WatchSource:0}: Error finding container 45aabeb4645b15599906063eb5eb9b41a2f90947ea792911fac81b0985b3afbd: Status 404 returned error can't find the container with id 45aabeb4645b15599906063eb5eb9b41a2f90947ea792911fac81b0985b3afbd
Oct 31 01:42:14.992285 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.992246    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84dc6d9888-zklck" event={"ID":"dc6371d3-66fb-4c1e-86a8-ddfd27197a14","Type":"ContainerStarted","Data":"3c519b04ec9513df791d4ca2ac3e93be0a60da6c49226a94b3b9d8e7d94c3ca6"}
Oct 31 01:42:14.992536 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.992306    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84dc6d9888-zklck" event={"ID":"dc6371d3-66fb-4c1e-86a8-ddfd27197a14","Type":"ContainerStarted","Data":"abd3c33da6c070e7061f5d8a433f16f45c0b17f17737e6fac4be1129ab90dc17"}
Oct 31 01:42:14.992536 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.992477    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-84dc6d9888-zklck"
Oct 31 01:42:14.994293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.994256    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv" event={"ID":"4ad91b85-a022-47b9-9872-4601a6b3215b","Type":"ContainerStarted","Data":"48b845e6fe4ceea3a802e712d94281997cdf3f6002b9d14f8ca00b788bd2181c"}
Oct 31 01:42:14.994402 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.994306    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv" event={"ID":"4ad91b85-a022-47b9-9872-4601a6b3215b","Type":"ContainerStarted","Data":"45aabeb4645b15599906063eb5eb9b41a2f90947ea792911fac81b0985b3afbd"}
Oct 31 01:42:14.994402 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:14.994335    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv"
Oct 31 01:42:15.021480 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:15.021371    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-84dc6d9888-zklck" podStartSLOduration=2.021349997 podStartE2EDuration="2.021349997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:15.017979238 +0000 UTC m=+1291.021693530" watchObservedRunningTime="2026-10-31 01:42:15.021349997 +0000 UTC m=+1291.025064286"
Oct 31 01:42:19.568926 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.568855    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv" podStartSLOduration=6.568830766 podStartE2EDuration="6.568830766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:15.047039114 +0000 UTC m=+1291.050753406" watchObservedRunningTime="2026-10-31 01:42:19.568830766 +0000 UTC m=+1295.572545057"
Oct 31 01:42:19.572126 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.570818    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"]
Oct 31 01:42:19.576059 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.576027    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.586269 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.585320    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"]
Oct 31 01:42:19.673182 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.673143    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.673368 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.673203    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.673368 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.673302    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.673368 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.673340    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d8cg\" (UniqueName: \"kubernetes.io/projected/0bcd9014-81f9-45a5-a32f-8cbe21750e80-kube-api-access-2d8cg\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.673522 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.673507    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.673565 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.673538    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.673622 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.673607    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.673659 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.673628    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.673693 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.673655    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.774995 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.774957    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.774995 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.774996    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.775209 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.775024    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.775209 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.775082    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.775284 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.775240    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.775325 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.775297    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.775370 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.775338    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d8cg\" (UniqueName: \"kubernetes.io/projected/0bcd9014-81f9-45a5-a32f-8cbe21750e80-kube-api-access-2d8cg\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.775476 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.775460    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.775564 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.775490    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.775564 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.775519    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.775665 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.775560    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.775716 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.775676    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.775860 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.775831    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.776241 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.776220    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.777419 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.777393    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.777702 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.777686    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.788299 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.788276    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.789471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.789434    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d8cg\" (UniqueName: \"kubernetes.io/projected/0bcd9014-81f9-45a5-a32f-8cbe21750e80-kube-api-access-2d8cg\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:19.895494 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:19.895155    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:20.228798 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:20.228761    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"]
Oct 31 01:42:20.236171 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:42:20.236140    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bcd9014_81f9_45a5_a32f_8cbe21750e80.slice/crio-347527c774a5551380d514cee55b72e1b74a4ef6d3e154f35b8dc21cc853c8d7 WatchSource:0}: Error finding container 347527c774a5551380d514cee55b72e1b74a4ef6d3e154f35b8dc21cc853c8d7: Status 404 returned error can't find the container with id 347527c774a5551380d514cee55b72e1b74a4ef6d3e154f35b8dc21cc853c8d7
Oct 31 01:42:20.240090 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:20.240047    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:42:20.240215 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:20.240141    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:42:20.240215 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:20.240189    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:42:21.049480 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:21.049416    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql" event={"ID":"0bcd9014-81f9-45a5-a32f-8cbe21750e80","Type":"ContainerStarted","Data":"ae5731e27b45330d7d678aca88b17475a6a2822c54aa67487dfa9c6e976647bb"}
Oct 31 01:42:21.049970 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:21.049528    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql" event={"ID":"0bcd9014-81f9-45a5-a32f-8cbe21750e80","Type":"ContainerStarted","Data":"347527c774a5551380d514cee55b72e1b74a4ef6d3e154f35b8dc21cc853c8d7"}
Oct 31 01:42:21.080262 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:21.080204    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql" podStartSLOduration=2.08018844 podStartE2EDuration="2.08018844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:21.077944445 +0000 UTC m=+1297.081658737" watchObservedRunningTime="2026-10-31 01:42:21.08018844 +0000 UTC m=+1297.083902787"
Oct 31 01:42:21.895770 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:21.895732    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:21.897738 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:21.897712    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:42:21.897877 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:21.897764    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql" podUID="0bcd9014-81f9-45a5-a32f-8cbe21750e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 31 01:42:22.896391 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:22.896352    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:42:22.896841 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:22.896421    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql" podUID="0bcd9014-81f9-45a5-a32f-8cbe21750e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 31 01:42:23.895828 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:23.895792    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:42:23.896010 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:23.895853    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql" podUID="0bcd9014-81f9-45a5-a32f-8cbe21750e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 31 01:42:24.896021 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:24.895985    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:42:24.896567 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:24.896041    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql" podUID="0bcd9014-81f9-45a5-a32f-8cbe21750e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 31 01:42:25.896412 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:25.896373    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:42:25.896856 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:25.896442    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql" podUID="0bcd9014-81f9-45a5-a32f-8cbe21750e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 31 01:42:26.025586 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:26.025550    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv"
Oct 31 01:42:26.026039 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:26.026014    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-84dc6d9888-zklck"
Oct 31 01:42:26.079428 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:26.079393    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-84dc6d9888-zklck"]
Oct 31 01:42:26.896556 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:26.896522    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:42:26.896965 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:26.896590    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql" podUID="0bcd9014-81f9-45a5-a32f-8cbe21750e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 31 01:42:27.902574 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:27.902529    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:27.903435 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:27.903409    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:27.904489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:27.904463    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:42:28.577087 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:42:28.577047    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:42:31.081765 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:31.081718    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-84dc6d9888-zklck" podUID="dc6371d3-66fb-4c1e-86a8-ddfd27197a14" containerName="payload-processing" containerID="cri-o://3c519b04ec9513df791d4ca2ac3e93be0a60da6c49226a94b3b9d8e7d94c3ca6" gracePeriod=25
Oct 31 01:42:31.331358 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:31.331330    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84dc6d9888-zklck"
Oct 31 01:42:31.469242 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:31.469210    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-routing-overlay-volume\") pod \"dc6371d3-66fb-4c1e-86a8-ddfd27197a14\" (UID: \"dc6371d3-66fb-4c1e-86a8-ddfd27197a14\") "
Oct 31 01:42:31.469426 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:31.469367    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-kube-api-access-9jd9z\" (UniqueName: \"kubernetes.io/projected/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-kube-api-access-9jd9z\") pod \"dc6371d3-66fb-4c1e-86a8-ddfd27197a14\" (UID: \"dc6371d3-66fb-4c1e-86a8-ddfd27197a14\") "
Oct 31 01:42:31.469426 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:31.469418    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-plugins-config-volume\") pod \"dc6371d3-66fb-4c1e-86a8-ddfd27197a14\" (UID: \"dc6371d3-66fb-4c1e-86a8-ddfd27197a14\") "
Oct 31 01:42:31.469545 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:31.469438    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-provider-credentials\") pod \"dc6371d3-66fb-4c1e-86a8-ddfd27197a14\" (UID: \"dc6371d3-66fb-4c1e-86a8-ddfd27197a14\") "
Oct 31 01:42:31.469653 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:31.469627    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-routing-overlay-volume" pod "dc6371d3-66fb-4c1e-86a8-ddfd27197a14" (UID: "dc6371d3-66fb-4c1e-86a8-ddfd27197a14"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:42:31.469818 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:31.469791    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-plugins-config-volume" pod "dc6371d3-66fb-4c1e-86a8-ddfd27197a14" (UID: "dc6371d3-66fb-4c1e-86a8-ddfd27197a14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:42:31.469948 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:31.469893    5650 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-routing-overlay-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:31.469948 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:31.469921    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:31.471683 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:31.471659    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-kube-api-access-9jd9z" pod "dc6371d3-66fb-4c1e-86a8-ddfd27197a14" (UID: "dc6371d3-66fb-4c1e-86a8-ddfd27197a14"). InnerVolumeSpecName "kube-api-access-9jd9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:31.471784 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:31.471746    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-provider-credentials" pod "dc6371d3-66fb-4c1e-86a8-ddfd27197a14" (UID: "dc6371d3-66fb-4c1e-86a8-ddfd27197a14"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:31.571556 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:31.571517    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jd9z\" (UniqueName: \"kubernetes.io/projected/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-kube-api-access-9jd9z\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:31.571556 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:31.571549    5650 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc6371d3-66fb-4c1e-86a8-ddfd27197a14-provider-credentials\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:32.125846 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:32.125813    5650 generic.go:356] "Generic (PLEG): container finished" podID="dc6371d3-66fb-4c1e-86a8-ddfd27197a14" containerID="3c519b04ec9513df791d4ca2ac3e93be0a60da6c49226a94b3b9d8e7d94c3ca6" exitCode=0
Oct 31 01:42:32.126241 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:32.125864    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84dc6d9888-zklck"
Oct 31 01:42:32.126241 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:32.125894    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84dc6d9888-zklck" event={"ID":"dc6371d3-66fb-4c1e-86a8-ddfd27197a14","Type":"ContainerDied","Data":"3c519b04ec9513df791d4ca2ac3e93be0a60da6c49226a94b3b9d8e7d94c3ca6"}
Oct 31 01:42:32.126241 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:32.125949    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84dc6d9888-zklck" event={"ID":"dc6371d3-66fb-4c1e-86a8-ddfd27197a14","Type":"ContainerDied","Data":"abd3c33da6c070e7061f5d8a433f16f45c0b17f17737e6fac4be1129ab90dc17"}
Oct 31 01:42:32.126241 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:32.125974    5650 scope.go:122] "RemoveContainer" containerID="3c519b04ec9513df791d4ca2ac3e93be0a60da6c49226a94b3b9d8e7d94c3ca6"
Oct 31 01:42:32.136180 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:32.136161    5650 scope.go:122] "RemoveContainer" containerID="3c519b04ec9513df791d4ca2ac3e93be0a60da6c49226a94b3b9d8e7d94c3ca6"
Oct 31 01:42:32.136485 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:42:32.136462    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c519b04ec9513df791d4ca2ac3e93be0a60da6c49226a94b3b9d8e7d94c3ca6\": container with ID starting with 3c519b04ec9513df791d4ca2ac3e93be0a60da6c49226a94b3b9d8e7d94c3ca6 not found: ID does not exist" containerID="3c519b04ec9513df791d4ca2ac3e93be0a60da6c49226a94b3b9d8e7d94c3ca6"
Oct 31 01:42:32.136532 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:32.136494    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c519b04ec9513df791d4ca2ac3e93be0a60da6c49226a94b3b9d8e7d94c3ca6"} err="failed to get container status \"3c519b04ec9513df791d4ca2ac3e93be0a60da6c49226a94b3b9d8e7d94c3ca6\": rpc error: code = NotFound desc = could not find container \"3c519b04ec9513df791d4ca2ac3e93be0a60da6c49226a94b3b9d8e7d94c3ca6\": container with ID starting with 3c519b04ec9513df791d4ca2ac3e93be0a60da6c49226a94b3b9d8e7d94c3ca6 not found: ID does not exist"
Oct 31 01:42:32.147481 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:32.147435    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-84dc6d9888-zklck"]
Oct 31 01:42:32.156884 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:32.156855    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-84dc6d9888-zklck"]
Oct 31 01:42:34.117767 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:34.117733    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc6371d3-66fb-4c1e-86a8-ddfd27197a14" path="/var/lib/kubelet/pods/dc6371d3-66fb-4c1e-86a8-ddfd27197a14/volumes"
Oct 31 01:42:37.154468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:37.153860    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf"]
Oct 31 01:42:37.165469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:37.160816    5650 cpu_manager.go:414] "containerMap: removing container" podUID="dc6371d3-66fb-4c1e-86a8-ddfd27197a14" containerName="payload-processing"
Oct 31 01:42:37.165469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:37.160843    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6371d3-66fb-4c1e-86a8-ddfd27197a14" containerName="payload-processing"
Oct 31 01:42:37.165469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:37.161020    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6371d3-66fb-4c1e-86a8-ddfd27197a14" containerName="payload-processing"
Oct 31 01:42:37.175765 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:37.175730    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf"
Oct 31 01:42:37.273223 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:37.263900    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50b155fa-11e1-42b6-b2a5-debe74cb96ef-provider-credentials\") pod \"payload-processing-external-model-7f9f9f6597-wbzjf\" (UID: \"50b155fa-11e1-42b6-b2a5-debe74cb96ef\") " pod="models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf"
Oct 31 01:42:37.273223 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:37.263974    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50b155fa-11e1-42b6-b2a5-debe74cb96ef-plugins-config-volume\") pod \"payload-processing-external-model-7f9f9f6597-wbzjf\" (UID: \"50b155fa-11e1-42b6-b2a5-debe74cb96ef\") " pod="models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf"
Oct 31 01:42:37.273223 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:37.263995    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50b155fa-11e1-42b6-b2a5-debe74cb96ef-routing-overlay-volume\") pod \"payload-processing-external-model-7f9f9f6597-wbzjf\" (UID: \"50b155fa-11e1-42b6-b2a5-debe74cb96ef\") " pod="models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf"
Oct 31 01:42:37.273223 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:37.264051    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8g2w\" (UniqueName: \"kubernetes.io/projected/50b155fa-11e1-42b6-b2a5-debe74cb96ef-kube-api-access-c8g2w\") pod \"payload-processing-external-model-7f9f9f6597-wbzjf\" (UID: \"50b155fa-11e1-42b6-b2a5-debe74cb96ef\") " pod="models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf"
Oct 31 01:42:37.291833 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:37.291585    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf"]
Oct 31 01:42:37.367994 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:37.366840    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50b155fa-11e1-42b6-b2a5-debe74cb96ef-provider-credentials\") pod \"payload-processing-external-model-7f9f9f6597-wbzjf\" (UID: \"50b155fa-11e1-42b6-b2a5-debe74cb96ef\") " pod="models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf"
Oct 31 01:42:37.368326 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:37.368298    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50b155fa-11e1-42b6-b2a5-debe74cb96ef-plugins-config-volume\") pod \"payload-processing-external-model-7f9f9f6597-wbzjf\" (UID: \"50b155fa-11e1-42b6-b2a5-debe74cb96ef\") " pod="models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf"
Oct 31 01:42:37.370527 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:37.369336    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50b155fa-11e1-42b6-b2a5-debe74cb96ef-routing-overlay-volume\") pod \"payload-processing-external-model-7f9f9f6597-wbzjf\" (UID: \"50b155fa-11e1-42b6-b2a5-debe74cb96ef\") " pod="models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf"
Oct 31 01:42:37.372829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:37.369269    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50b155fa-11e1-42b6-b2a5-debe74cb96ef-plugins-config-volume\") pod \"payload-processing-external-model-7f9f9f6597-wbzjf\" (UID: \"50b155fa-11e1-42b6-b2a5-debe74cb96ef\") " pod="models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf"
Oct 31 01:42:37.372829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:37.372687    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50b155fa-11e1-42b6-b2a5-debe74cb96ef-routing-overlay-volume\") pod \"payload-processing-external-model-7f9f9f6597-wbzjf\" (UID: \"50b155fa-11e1-42b6-b2a5-debe74cb96ef\") " pod="models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf"
Oct 31 01:42:37.373138 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:37.373110    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8g2w\" (UniqueName: \"kubernetes.io/projected/50b155fa-11e1-42b6-b2a5-debe74cb96ef-kube-api-access-c8g2w\") pod \"payload-processing-external-model-7f9f9f6597-wbzjf\" (UID: \"50b155fa-11e1-42b6-b2a5-debe74cb96ef\") " pod="models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf"
Oct 31 01:42:37.381459 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:37.381295    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50b155fa-11e1-42b6-b2a5-debe74cb96ef-provider-credentials\") pod \"payload-processing-external-model-7f9f9f6597-wbzjf\" (UID: \"50b155fa-11e1-42b6-b2a5-debe74cb96ef\") " pod="models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf"
Oct 31 01:42:37.387769 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:37.387715    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8g2w\" (UniqueName: \"kubernetes.io/projected/50b155fa-11e1-42b6-b2a5-debe74cb96ef-kube-api-access-c8g2w\") pod \"payload-processing-external-model-7f9f9f6597-wbzjf\" (UID: \"50b155fa-11e1-42b6-b2a5-debe74cb96ef\") " pod="models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf"
Oct 31 01:42:37.593171 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:37.593123    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf"
Oct 31 01:42:37.956328 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:37.940873    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf"]
Oct 31 01:42:37.956328 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:42:37.956267    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b155fa_11e1_42b6_b2a5_debe74cb96ef.slice/crio-eaddacc526950477e4f32fbdf0e852073f84e434d77ea987f5fa858df23205a0 WatchSource:0}: Error finding container eaddacc526950477e4f32fbdf0e852073f84e434d77ea987f5fa858df23205a0: Status 404 returned error can't find the container with id eaddacc526950477e4f32fbdf0e852073f84e434d77ea987f5fa858df23205a0
Oct 31 01:42:38.283570 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:38.282928    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf"]
Oct 31 01:42:38.377676 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:38.377629    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf" event={"ID":"50b155fa-11e1-42b6-b2a5-debe74cb96ef","Type":"ContainerStarted","Data":"e9b207c0275ef6d6749a32f1375086e05e4e6129e6e957b7eedf07af425c55cc"}
Oct 31 01:42:38.378005 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:38.377975    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf" event={"ID":"50b155fa-11e1-42b6-b2a5-debe74cb96ef","Type":"ContainerStarted","Data":"eaddacc526950477e4f32fbdf0e852073f84e434d77ea987f5fa858df23205a0"}
Oct 31 01:42:38.379335 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:38.379313    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf"
Oct 31 01:42:38.402012 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:42:38.401958    5650 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-735cc15d15ca: secret "agc-cred-735cc15d15ca" not found
Oct 31 01:42:38.402252 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:42:38.402236    5650 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-efc045041ce5: secret "agc-cred-efc045041ce5" not found
Oct 31 01:42:38.402357 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:42:38.402344    5650 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf: [secret "agc-cred-735cc15d15ca" not found, secret "agc-cred-efc045041ce5" not found]
Oct 31 01:42:38.402533 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:42:38.402518    5650 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50b155fa-11e1-42b6-b2a5-debe74cb96ef-provider-credentials podName:50b155fa-11e1-42b6-b2a5-debe74cb96ef nodeName:}" failed. No retries permitted until 2026-10-31 01:42:38.902494376 +0000 UTC m=+1314.906208662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/50b155fa-11e1-42b6-b2a5-debe74cb96ef-provider-credentials") pod "payload-processing-external-model-7f9f9f6597-wbzjf" (UID: "50b155fa-11e1-42b6-b2a5-debe74cb96ef") : [secret "agc-cred-735cc15d15ca" not found, secret "agc-cred-efc045041ce5" not found]
Oct 31 01:42:38.411561 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:38.411509    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf" podStartSLOduration=1.411490462 podStartE2EDuration="1.411490462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:38.41060138 +0000 UTC m=+1314.414315671" watchObservedRunningTime="2026-10-31 01:42:38.411490462 +0000 UTC m=+1314.415204754"
Oct 31 01:42:38.909581 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:42:38.909546    5650 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-735cc15d15ca: secret "agc-cred-735cc15d15ca" not found
Oct 31 01:42:38.909581 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:42:38.909581    5650 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-efc045041ce5: secret "agc-cred-efc045041ce5" not found
Oct 31 01:42:38.909581 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:42:38.909591    5650 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf: [secret "agc-cred-735cc15d15ca" not found, secret "agc-cred-efc045041ce5" not found]
Oct 31 01:42:38.909812 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:42:38.909647    5650 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50b155fa-11e1-42b6-b2a5-debe74cb96ef-provider-credentials podName:50b155fa-11e1-42b6-b2a5-debe74cb96ef nodeName:}" failed. No retries permitted until 2026-10-31 01:42:39.909632129 +0000 UTC m=+1315.913346398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/50b155fa-11e1-42b6-b2a5-debe74cb96ef-provider-credentials") pod "payload-processing-external-model-7f9f9f6597-wbzjf" (UID: "50b155fa-11e1-42b6-b2a5-debe74cb96ef") : [secret "agc-cred-735cc15d15ca" not found, secret "agc-cred-efc045041ce5" not found]
Oct 31 01:42:39.660407 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:39.660374    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"]
Oct 31 01:42:39.664746 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:39.664722    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:42:39.672191 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:39.672166    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6ae519\"/\"distinct-w2-6ae519-kserve-self-signed-certs\""
Oct 31 01:42:39.675300 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:39.675272    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"]
Oct 31 01:42:39.717681 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:39.717646    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-home\") pod \"distinct-w2-6ae519-kserve-5f68644bdb-6r98n\" (UID: \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:42:39.717886 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:39.717706    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-dshm\") pod \"distinct-w2-6ae519-kserve-5f68644bdb-6r98n\" (UID: \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:42:39.717886 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:39.717794    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-tmp-dir\") pod \"distinct-w2-6ae519-kserve-5f68644bdb-6r98n\" (UID: \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:42:39.717990 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:39.717899    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210cc1c7-6a02-495c-89f1-6dc889f8bf11-tls-certs\") pod \"distinct-w2-6ae519-kserve-5f68644bdb-6r98n\" (UID: \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:42:39.718052 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:39.718014    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-model-cache\") pod \"distinct-w2-6ae519-kserve-5f68644bdb-6r98n\" (UID: \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:42:39.718107 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:39.718069    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42dwd\" (UniqueName: \"kubernetes.io/projected/210cc1c7-6a02-495c-89f1-6dc889f8bf11-kube-api-access-42dwd\") pod \"distinct-w2-6ae519-kserve-5f68644bdb-6r98n\" (UID: \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:42:39.819241 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:39.819205    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-model-cache\") pod \"distinct-w2-6ae519-kserve-5f68644bdb-6r98n\" (UID: \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:42:39.819428 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:39.819256    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42dwd\" (UniqueName: \"kubernetes.io/projected/210cc1c7-6a02-495c-89f1-6dc889f8bf11-kube-api-access-42dwd\") pod \"distinct-w2-6ae519-kserve-5f68644bdb-6r98n\" (UID: \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:42:39.819428 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:39.819320    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-home\") pod \"distinct-w2-6ae519-kserve-5f68644bdb-6r98n\" (UID: \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:42:39.819428 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:39.819376    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-dshm\") pod \"distinct-w2-6ae519-kserve-5f68644bdb-6r98n\" (UID: \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:42:39.819428 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:39.819415    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-tmp-dir\") pod \"distinct-w2-6ae519-kserve-5f68644bdb-6r98n\" (UID: \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:42:39.819656 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:39.819475    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210cc1c7-6a02-495c-89f1-6dc889f8bf11-tls-certs\") pod \"distinct-w2-6ae519-kserve-5f68644bdb-6r98n\" (UID: \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:42:39.819884 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:39.819854    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-home\") pod \"distinct-w2-6ae519-kserve-5f68644bdb-6r98n\" (UID: \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:42:39.819958 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:39.819936    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-tmp-dir\") pod \"distinct-w2-6ae519-kserve-5f68644bdb-6r98n\" (UID: \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:42:39.820009 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:39.819982    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-model-cache\") pod \"distinct-w2-6ae519-kserve-5f68644bdb-6r98n\" (UID: \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:42:39.822149 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:39.821871    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-dshm\") pod \"distinct-w2-6ae519-kserve-5f68644bdb-6r98n\" (UID: \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:42:39.822484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:39.822334    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210cc1c7-6a02-495c-89f1-6dc889f8bf11-tls-certs\") pod \"distinct-w2-6ae519-kserve-5f68644bdb-6r98n\" (UID: \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:42:39.832285 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:39.832259    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42dwd\" (UniqueName: \"kubernetes.io/projected/210cc1c7-6a02-495c-89f1-6dc889f8bf11-kube-api-access-42dwd\") pod \"distinct-w2-6ae519-kserve-5f68644bdb-6r98n\" (UID: \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:42:39.921331 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:42:39.921247    5650 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-735cc15d15ca: secret "agc-cred-735cc15d15ca" not found
Oct 31 01:42:39.921331 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:42:39.921295    5650 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-efc045041ce5: secret "agc-cred-efc045041ce5" not found
Oct 31 01:42:39.921331 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:42:39.921308    5650 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf: [secret "agc-cred-735cc15d15ca" not found, secret "agc-cred-efc045041ce5" not found]
Oct 31 01:42:39.921571 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:42:39.921378    5650 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50b155fa-11e1-42b6-b2a5-debe74cb96ef-provider-credentials podName:50b155fa-11e1-42b6-b2a5-debe74cb96ef nodeName:}" failed. No retries permitted until 2026-10-31 01:42:41.921355759 +0000 UTC m=+1317.925070031 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/50b155fa-11e1-42b6-b2a5-debe74cb96ef-provider-credentials") pod "payload-processing-external-model-7f9f9f6597-wbzjf" (UID: "50b155fa-11e1-42b6-b2a5-debe74cb96ef") : [secret "agc-cred-735cc15d15ca" not found, secret "agc-cred-efc045041ce5" not found]
Oct 31 01:42:39.980383 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:39.980344    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:42:40.263292 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:40.263261    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"]
Oct 31 01:42:40.264477 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:42:40.264433    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod210cc1c7_6a02_495c_89f1_6dc889f8bf11.slice/crio-2f8c1ef11d4dcc1b19aed50a0e610b0afb50779e96df34c7bdecdc26c4586188 WatchSource:0}: Error finding container 2f8c1ef11d4dcc1b19aed50a0e610b0afb50779e96df34c7bdecdc26c4586188: Status 404 returned error can't find the container with id 2f8c1ef11d4dcc1b19aed50a0e610b0afb50779e96df34c7bdecdc26c4586188
Oct 31 01:42:40.392378 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:40.392345    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" event={"ID":"210cc1c7-6a02-495c-89f1-6dc889f8bf11","Type":"ContainerStarted","Data":"bc6e7782cd99eccb1f87e8dae71dfee672433454fef75c490095dfcc5ba346d9"}
Oct 31 01:42:40.392570 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:40.392392    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" event={"ID":"210cc1c7-6a02-495c-89f1-6dc889f8bf11","Type":"ContainerStarted","Data":"2f8c1ef11d4dcc1b19aed50a0e610b0afb50779e96df34c7bdecdc26c4586188"}
Oct 31 01:42:40.413843 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:40.413793    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podStartSLOduration=1.413778081 podStartE2EDuration="1.413778081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:40.412751353 +0000 UTC m=+1316.416465644" watchObservedRunningTime="2026-10-31 01:42:40.413778081 +0000 UTC m=+1316.417492412"
Oct 31 01:42:40.980900 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:40.980858    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:42:41.944346 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:42:41.944314    5650 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-735cc15d15ca: secret "agc-cred-735cc15d15ca" not found
Oct 31 01:42:41.944346 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:42:41.944350    5650 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-efc045041ce5: secret "agc-cred-efc045041ce5" not found
Oct 31 01:42:41.944655 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:42:41.944361    5650 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf: [secret "agc-cred-735cc15d15ca" not found, secret "agc-cred-efc045041ce5" not found]
Oct 31 01:42:41.944655 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:42:41.944422    5650 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50b155fa-11e1-42b6-b2a5-debe74cb96ef-provider-credentials podName:50b155fa-11e1-42b6-b2a5-debe74cb96ef nodeName:}" failed. No retries permitted until 2026-10-31 01:42:45.94440466 +0000 UTC m=+1321.948118948 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/50b155fa-11e1-42b6-b2a5-debe74cb96ef-provider-credentials") pod "payload-processing-external-model-7f9f9f6597-wbzjf" (UID: "50b155fa-11e1-42b6-b2a5-debe74cb96ef") : [secret "agc-cred-735cc15d15ca" not found, secret "agc-cred-efc045041ce5" not found]
Oct 31 01:42:44.385227 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:44.385138    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf" podUID="50b155fa-11e1-42b6-b2a5-debe74cb96ef" containerName="payload-processing" containerID="cri-o://e9b207c0275ef6d6749a32f1375086e05e4e6129e6e957b7eedf07af425c55cc" gracePeriod=25
Oct 31 01:42:44.642438 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:44.642412    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf"
Oct 31 01:42:44.778179 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:44.778143    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b155fa-11e1-42b6-b2a5-debe74cb96ef-provider-credentials\" (UniqueName: \"kubernetes.io/projected/50b155fa-11e1-42b6-b2a5-debe74cb96ef-provider-credentials\") pod \"50b155fa-11e1-42b6-b2a5-debe74cb96ef\" (UID: \"50b155fa-11e1-42b6-b2a5-debe74cb96ef\") "
Oct 31 01:42:44.778358 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:44.778231    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b155fa-11e1-42b6-b2a5-debe74cb96ef-kube-api-access-c8g2w\" (UniqueName: \"kubernetes.io/projected/50b155fa-11e1-42b6-b2a5-debe74cb96ef-kube-api-access-c8g2w\") pod \"50b155fa-11e1-42b6-b2a5-debe74cb96ef\" (UID: \"50b155fa-11e1-42b6-b2a5-debe74cb96ef\") "
Oct 31 01:42:44.778481 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:44.778426    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b155fa-11e1-42b6-b2a5-debe74cb96ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50b155fa-11e1-42b6-b2a5-debe74cb96ef-plugins-config-volume\") pod \"50b155fa-11e1-42b6-b2a5-debe74cb96ef\" (UID: \"50b155fa-11e1-42b6-b2a5-debe74cb96ef\") "
Oct 31 01:42:44.778535 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:44.778496    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b155fa-11e1-42b6-b2a5-debe74cb96ef-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50b155fa-11e1-42b6-b2a5-debe74cb96ef-routing-overlay-volume\") pod \"50b155fa-11e1-42b6-b2a5-debe74cb96ef\" (UID: \"50b155fa-11e1-42b6-b2a5-debe74cb96ef\") "
Oct 31 01:42:44.778837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:44.778809    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b155fa-11e1-42b6-b2a5-debe74cb96ef-routing-overlay-volume" pod "50b155fa-11e1-42b6-b2a5-debe74cb96ef" (UID: "50b155fa-11e1-42b6-b2a5-debe74cb96ef"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:42:44.778916 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:44.778846    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b155fa-11e1-42b6-b2a5-debe74cb96ef-plugins-config-volume" pod "50b155fa-11e1-42b6-b2a5-debe74cb96ef" (UID: "50b155fa-11e1-42b6-b2a5-debe74cb96ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:42:44.778959 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:44.778939    5650 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50b155fa-11e1-42b6-b2a5-debe74cb96ef-routing-overlay-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:44.780469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:44.780431    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b155fa-11e1-42b6-b2a5-debe74cb96ef-provider-credentials" pod "50b155fa-11e1-42b6-b2a5-debe74cb96ef" (UID: "50b155fa-11e1-42b6-b2a5-debe74cb96ef"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:44.780522 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:44.780442    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b155fa-11e1-42b6-b2a5-debe74cb96ef-kube-api-access-c8g2w" pod "50b155fa-11e1-42b6-b2a5-debe74cb96ef" (UID: "50b155fa-11e1-42b6-b2a5-debe74cb96ef"). InnerVolumeSpecName "kube-api-access-c8g2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:44.880030 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:44.879943    5650 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50b155fa-11e1-42b6-b2a5-debe74cb96ef-provider-credentials\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:44.880030 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:44.879977    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8g2w\" (UniqueName: \"kubernetes.io/projected/50b155fa-11e1-42b6-b2a5-debe74cb96ef-kube-api-access-c8g2w\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:44.880030 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:44.879988    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50b155fa-11e1-42b6-b2a5-debe74cb96ef-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:45.063678 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.063637    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"]
Oct 31 01:42:45.064230 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.064211    5650 cpu_manager.go:414] "containerMap: removing container" podUID="50b155fa-11e1-42b6-b2a5-debe74cb96ef" containerName="payload-processing"
Oct 31 01:42:45.064324 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.064231    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b155fa-11e1-42b6-b2a5-debe74cb96ef" containerName="payload-processing"
Oct 31 01:42:45.064382 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.064368    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b155fa-11e1-42b6-b2a5-debe74cb96ef" containerName="payload-processing"
Oct 31 01:42:45.067798 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.067775    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.086359 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.086327    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"]
Oct 31 01:42:45.100685 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.100647    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"]
Oct 31 01:42:45.105497 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.105435    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.120272 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.120239    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"]
Oct 31 01:42:45.184643 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.184552    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdcc2\" (UniqueName: \"kubernetes.io/projected/ced27998-939e-4734-9904-18c4dc0b1170-kube-api-access-xdcc2\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.184643 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.184598    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.184643 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.184628    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced27998-939e-4734-9904-18c4dc0b1170-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.184929 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.184733    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced27998-939e-4734-9904-18c4dc0b1170-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.184929 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.184783    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced27998-939e-4734-9904-18c4dc0b1170-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.184929 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.184816    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced27998-939e-4734-9904-18c4dc0b1170-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.184929 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.184842    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced27998-939e-4734-9904-18c4dc0b1170-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.184929 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.184896    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.185183 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.184931    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52s84\" (UniqueName: \"kubernetes.io/projected/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-kube-api-access-52s84\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.185183 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.185065    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced27998-939e-4734-9904-18c4dc0b1170-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.185183 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.185132    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.185183 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.185166    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.185372 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.185200    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.185372 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.185296    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced27998-939e-4734-9904-18c4dc0b1170-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.185485 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.185369    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.185485 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.185396    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.185485 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.185423    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced27998-939e-4734-9904-18c4dc0b1170-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.185637 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.185485    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.287115 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.287074    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced27998-939e-4734-9904-18c4dc0b1170-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.287115 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.287122    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.287347 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.287144    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.287347 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.287167    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced27998-939e-4734-9904-18c4dc0b1170-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.287347 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.287185    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.287347 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.287309    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdcc2\" (UniqueName: \"kubernetes.io/projected/ced27998-939e-4734-9904-18c4dc0b1170-kube-api-access-xdcc2\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.287594 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.287384    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.287594 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.287429    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced27998-939e-4734-9904-18c4dc0b1170-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.287594 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.287496    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced27998-939e-4734-9904-18c4dc0b1170-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.287594 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.287541    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced27998-939e-4734-9904-18c4dc0b1170-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.287808 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.287598    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced27998-939e-4734-9904-18c4dc0b1170-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.287808 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.287619    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.287808 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.287654    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced27998-939e-4734-9904-18c4dc0b1170-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.287808 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.287737    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced27998-939e-4734-9904-18c4dc0b1170-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.288019 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.287936    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced27998-939e-4734-9904-18c4dc0b1170-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.288074 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.288056    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.288136 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.288097    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52s84\" (UniqueName: \"kubernetes.io/projected/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-kube-api-access-52s84\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.288192 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.288127    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.288253 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.288208    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced27998-939e-4734-9904-18c4dc0b1170-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.288253 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.288220    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced27998-939e-4734-9904-18c4dc0b1170-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.288253 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.288057    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced27998-939e-4734-9904-18c4dc0b1170-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.288405 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.288299    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced27998-939e-4734-9904-18c4dc0b1170-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.288405 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.288310    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.288405 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.288327    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.288405 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.288392    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.288606 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.288427    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.288606 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.288582    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.288775 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.288753    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.289804 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.289768    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.289950 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.289922    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced27998-939e-4734-9904-18c4dc0b1170-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.290201 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.290186    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced27998-939e-4734-9904-18c4dc0b1170-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.290607 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.290588    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.300603 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.300572    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.300999 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.300891    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced27998-939e-4734-9904-18c4dc0b1170-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.301544 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.301520    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdcc2\" (UniqueName: \"kubernetes.io/projected/ced27998-939e-4734-9904-18c4dc0b1170-kube-api-access-xdcc2\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jpp7c\" (UID: \"ced27998-939e-4734-9904-18c4dc0b1170\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.308645 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.308619    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52s84\" (UniqueName: \"kubernetes.io/projected/b4ebd16b-3010-4a21-abd5-8b1f7e8c4905-kube-api-access-52s84\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zznwh\" (UID: \"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.383247 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.383208    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:45.426635 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.426433    5650 generic.go:356] "Generic (PLEG): container finished" podID="50b155fa-11e1-42b6-b2a5-debe74cb96ef" containerID="e9b207c0275ef6d6749a32f1375086e05e4e6129e6e957b7eedf07af425c55cc" exitCode=0
Oct 31 01:42:45.426635 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.426501    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf" event={"ID":"50b155fa-11e1-42b6-b2a5-debe74cb96ef","Type":"ContainerDied","Data":"e9b207c0275ef6d6749a32f1375086e05e4e6129e6e957b7eedf07af425c55cc"}
Oct 31 01:42:45.426635 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.426519    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf"
Oct 31 01:42:45.426635 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.426581    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf" event={"ID":"50b155fa-11e1-42b6-b2a5-debe74cb96ef","Type":"ContainerDied","Data":"eaddacc526950477e4f32fbdf0e852073f84e434d77ea987f5fa858df23205a0"}
Oct 31 01:42:45.426635 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.426608    5650 scope.go:122] "RemoveContainer" containerID="e9b207c0275ef6d6749a32f1375086e05e4e6129e6e957b7eedf07af425c55cc"
Oct 31 01:42:45.428846 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.428605    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:45.453241 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.450927    5650 scope.go:122] "RemoveContainer" containerID="e9b207c0275ef6d6749a32f1375086e05e4e6129e6e957b7eedf07af425c55cc"
Oct 31 01:42:45.453241 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:42:45.451512    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9b207c0275ef6d6749a32f1375086e05e4e6129e6e957b7eedf07af425c55cc\": container with ID starting with e9b207c0275ef6d6749a32f1375086e05e4e6129e6e957b7eedf07af425c55cc not found: ID does not exist" containerID="e9b207c0275ef6d6749a32f1375086e05e4e6129e6e957b7eedf07af425c55cc"
Oct 31 01:42:45.453241 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.451576    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9b207c0275ef6d6749a32f1375086e05e4e6129e6e957b7eedf07af425c55cc"} err="failed to get container status \"e9b207c0275ef6d6749a32f1375086e05e4e6129e6e957b7eedf07af425c55cc\": rpc error: code = NotFound desc = could not find container \"e9b207c0275ef6d6749a32f1375086e05e4e6129e6e957b7eedf07af425c55cc\": container with ID starting with e9b207c0275ef6d6749a32f1375086e05e4e6129e6e957b7eedf07af425c55cc not found: ID does not exist"
Oct 31 01:42:45.472419 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.472111    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf"]
Oct 31 01:42:45.472419 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.472178    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f9f9f6597-wbzjf"]
Oct 31 01:42:45.673357 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.673321    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"]
Oct 31 01:42:45.674364 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:42:45.674339    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced27998_939e_4734_9904_18c4dc0b1170.slice/crio-ead5624ed0ad0e6ea2b59b3d65bad504aaa1773a185d1ed9114fbac65e587ff8 WatchSource:0}: Error finding container ead5624ed0ad0e6ea2b59b3d65bad504aaa1773a185d1ed9114fbac65e587ff8: Status 404 returned error can't find the container with id ead5624ed0ad0e6ea2b59b3d65bad504aaa1773a185d1ed9114fbac65e587ff8
Oct 31 01:42:45.677407 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.677372    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:42:45.677516 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.677475    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:42:45.677564 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.677536    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:42:45.759253 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.759225    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"]
Oct 31 01:42:45.760953 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:42:45.760903    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ebd16b_3010_4a21_abd5_8b1f7e8c4905.slice/crio-ab56f4726a86f232abb1397d3962589087b0fb695ea42d623ad588509403fb2d WatchSource:0}: Error finding container ab56f4726a86f232abb1397d3962589087b0fb695ea42d623ad588509403fb2d: Status 404 returned error can't find the container with id ab56f4726a86f232abb1397d3962589087b0fb695ea42d623ad588509403fb2d
Oct 31 01:42:45.763659 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.763627    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:42:45.763770 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.763710    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:42:45.763770 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:45.763752    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:42:46.120528 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:46.120495    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50b155fa-11e1-42b6-b2a5-debe74cb96ef" path="/var/lib/kubelet/pods/50b155fa-11e1-42b6-b2a5-debe74cb96ef/volumes"
Oct 31 01:42:46.464903 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:46.464860    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c" event={"ID":"ced27998-939e-4734-9904-18c4dc0b1170","Type":"ContainerStarted","Data":"80135eef9542d12f845f3013caabf32f780c5a3d2f578ee0551639f004ffa3d7"}
Oct 31 01:42:46.465403 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:46.464944    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c" event={"ID":"ced27998-939e-4734-9904-18c4dc0b1170","Type":"ContainerStarted","Data":"ead5624ed0ad0e6ea2b59b3d65bad504aaa1773a185d1ed9114fbac65e587ff8"}
Oct 31 01:42:46.468938 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:46.468902    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh" event={"ID":"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905","Type":"ContainerStarted","Data":"d237395c1fdddd7a1057a04a870c719e1cc53dd43128c8261c582d413b1ed20a"}
Oct 31 01:42:46.469128 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:46.468962    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh" event={"ID":"b4ebd16b-3010-4a21-abd5-8b1f7e8c4905","Type":"ContainerStarted","Data":"ab56f4726a86f232abb1397d3962589087b0fb695ea42d623ad588509403fb2d"}
Oct 31 01:42:46.579127 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:46.578505    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c" podStartSLOduration=1.578486262 podStartE2EDuration="1.578486262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:46.530401199 +0000 UTC m=+1322.534115493" watchObservedRunningTime="2026-10-31 01:42:46.578486262 +0000 UTC m=+1322.582200553"
Oct 31 01:42:46.579768 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:46.579706    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh" podStartSLOduration=1.579691328 podStartE2EDuration="1.579691328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:46.577795729 +0000 UTC m=+1322.581510022" watchObservedRunningTime="2026-10-31 01:42:46.579691328 +0000 UTC m=+1322.583405620"
Oct 31 01:42:47.398470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:47.395927    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:47.431474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:47.427603    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 01:42:47.431474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:47.427667    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c" podUID="ced27998-939e-4734-9904-18c4dc0b1170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 01:42:47.461858 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:47.461821    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:47.525490 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:47.520988    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zznwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 01:42:47.525490 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:47.521053    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh" podUID="b4ebd16b-3010-4a21-abd5-8b1f7e8c4905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 01:42:48.384472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:48.384091    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 01:42:48.384472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:48.384153    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c" podUID="ced27998-939e-4734-9904-18c4dc0b1170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 01:42:48.430471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:48.429974    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zznwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 01:42:48.430471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:48.430053    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh" podUID="b4ebd16b-3010-4a21-abd5-8b1f7e8c4905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 01:42:48.430471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:48.430398    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 01:42:48.746642 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:48.746572    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:42:49.396471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:49.393744    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 01:42:49.396471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:49.393826    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c" podUID="ced27998-939e-4734-9904-18c4dc0b1170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 01:42:49.471471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:49.455883    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zznwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 01:42:49.471471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:49.455957    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh" podUID="b4ebd16b-3010-4a21-abd5-8b1f7e8c4905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 01:42:49.989473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:49.984981    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:42:50.386986 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:50.386893    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 01:42:50.387249 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:50.387217    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c" podUID="ced27998-939e-4734-9904-18c4dc0b1170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 01:42:50.433924 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:50.433875    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zznwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 01:42:50.435480 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:50.435424    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh" podUID="b4ebd16b-3010-4a21-abd5-8b1f7e8c4905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 01:42:50.505471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:50.503574    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:42:50.674548 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:50.665762    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv"]
Oct 31 01:42:51.015607 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:51.015559    5650 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": context deadline exceeded"
Oct 31 01:42:51.384737 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:51.384641    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 01:42:51.384737 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:51.384714    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c" podUID="ced27998-939e-4734-9904-18c4dc0b1170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 01:42:51.440471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:51.439909    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zznwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 01:42:51.440471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:51.439977    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh" podUID="b4ebd16b-3010-4a21-abd5-8b1f7e8c4905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 01:42:52.383993 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:52.383947    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 01:42:52.384400 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:52.384034    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c" podUID="ced27998-939e-4734-9904-18c4dc0b1170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 01:42:52.430430 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:52.430385    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zznwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 01:42:52.430646 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:52.430474    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh" podUID="b4ebd16b-3010-4a21-abd5-8b1f7e8c4905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 01:42:53.115297 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.115261    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"]
Oct 31 01:42:53.120752 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.120725    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.132424 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.132388    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"]
Oct 31 01:42:53.216771 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.216725    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.216980 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.216787    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f14608d5-81ae-4ba3-8986-a8e602befe12-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.216980 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.216938    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.216980 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.216966    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcghd\" (UniqueName: \"kubernetes.io/projected/f14608d5-81ae-4ba3-8986-a8e602befe12-kube-api-access-rcghd\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.217146 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.217039    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.217251 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.217219    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.217306 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.217264    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.217407 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.217390    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.217474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.217413    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.318980 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.318931    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.318980 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.318982    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcghd\" (UniqueName: \"kubernetes.io/projected/f14608d5-81ae-4ba3-8986-a8e602befe12-kube-api-access-rcghd\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.319256 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.319019    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.319256 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.319059    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.319256 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.319082    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.319256 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.319166    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.319256 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.319189    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.319256 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.319253    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.319678 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.319288    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f14608d5-81ae-4ba3-8986-a8e602befe12-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.319678 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.319439    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.319678 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.319562    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.319678 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.319604    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.319873 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.319733    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.320061 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.320038    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f14608d5-81ae-4ba3-8986-a8e602befe12-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.321920 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.321885    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.322167 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.322147    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.333568 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.333539    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.333677 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.333641    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcghd\" (UniqueName: \"kubernetes.io/projected/f14608d5-81ae-4ba3-8986-a8e602befe12-kube-api-access-rcghd\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.383837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.383738    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 01:42:53.383837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.383808    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c" podUID="ced27998-939e-4734-9904-18c4dc0b1170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 01:42:53.430342 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.430299    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zznwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 01:42:53.430788 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.430374    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh" podUID="b4ebd16b-3010-4a21-abd5-8b1f7e8c4905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 01:42:53.439913 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.439874    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:53.738846 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.738804    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"]
Oct 31 01:42:53.740794 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:42:53.740767    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14608d5_81ae_4ba3_8986_a8e602befe12.slice/crio-8bdc075509e508c2d8809b35bc2cae54d87e3c83a6dfe44e124ff921444bd5b0 WatchSource:0}: Error finding container 8bdc075509e508c2d8809b35bc2cae54d87e3c83a6dfe44e124ff921444bd5b0: Status 404 returned error can't find the container with id 8bdc075509e508c2d8809b35bc2cae54d87e3c83a6dfe44e124ff921444bd5b0
Oct 31 01:42:53.744243 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.744200    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:42:53.744334 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.744299    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:42:53.744377 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.744347    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:42:53.979574 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.978949    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h" event={"ID":"f14608d5-81ae-4ba3-8986-a8e602befe12","Type":"ContainerStarted","Data":"5f2902debda7847f750050fc5485f68e6a58eeaecc2a246cddf916fd5923ad9f"}
Oct 31 01:42:53.979574 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:53.979018    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h" event={"ID":"f14608d5-81ae-4ba3-8986-a8e602befe12","Type":"ContainerStarted","Data":"8bdc075509e508c2d8809b35bc2cae54d87e3c83a6dfe44e124ff921444bd5b0"}
Oct 31 01:42:54.006530 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:54.005576    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h" podStartSLOduration=1.005557589 podStartE2EDuration="1.005557589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:54.005017025 +0000 UTC m=+1330.008731318" watchObservedRunningTime="2026-10-31 01:42:54.005557589 +0000 UTC m=+1330.009271882"
Oct 31 01:42:54.443531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:54.443210    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:42:54.443531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:54.443467    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:42:54.444068 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:54.443526    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h" podUID="f14608d5-81ae-4ba3-8986-a8e602befe12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 01:42:55.407482 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:55.406591    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:42:55.407482 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:55.406662    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c" podUID="ced27998-939e-4734-9904-18c4dc0b1170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:42:55.438058 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:55.437570    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zznwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:42:55.438058 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:55.437657    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh" podUID="b4ebd16b-3010-4a21-abd5-8b1f7e8c4905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:42:55.443199 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:55.443162    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:42:55.443486 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:55.443437    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h" podUID="f14608d5-81ae-4ba3-8986-a8e602befe12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 01:42:55.676462 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:55.675387    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv" podUID="4ad91b85-a022-47b9-9872-4601a6b3215b" containerName="payload-processing" containerID="cri-o://48b845e6fe4ceea3a802e712d94281997cdf3f6002b9d14f8ca00b788bd2181c" gracePeriod=25
Oct 31 01:42:56.069073 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:56.069031    5650 generic.go:356] "Generic (PLEG): container finished" podID="4ad91b85-a022-47b9-9872-4601a6b3215b" containerID="48b845e6fe4ceea3a802e712d94281997cdf3f6002b9d14f8ca00b788bd2181c" exitCode=0
Oct 31 01:42:56.069958 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:56.069328    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv" event={"ID":"4ad91b85-a022-47b9-9872-4601a6b3215b","Type":"ContainerDied","Data":"48b845e6fe4ceea3a802e712d94281997cdf3f6002b9d14f8ca00b788bd2181c"}
Oct 31 01:42:56.200483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:56.197971    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv"
Oct 31 01:42:56.307538 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:56.304946    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad91b85-a022-47b9-9872-4601a6b3215b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad91b85-a022-47b9-9872-4601a6b3215b-plugins-config-volume\") pod \"4ad91b85-a022-47b9-9872-4601a6b3215b\" (UID: \"4ad91b85-a022-47b9-9872-4601a6b3215b\") "
Oct 31 01:42:56.307538 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:56.305033    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad91b85-a022-47b9-9872-4601a6b3215b-kube-api-access-b5sp5\" (UniqueName: \"kubernetes.io/projected/4ad91b85-a022-47b9-9872-4601a6b3215b-kube-api-access-b5sp5\") pod \"4ad91b85-a022-47b9-9872-4601a6b3215b\" (UID: \"4ad91b85-a022-47b9-9872-4601a6b3215b\") "
Oct 31 01:42:56.307538 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:56.305170    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad91b85-a022-47b9-9872-4601a6b3215b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ad91b85-a022-47b9-9872-4601a6b3215b-routing-overlay-volume\") pod \"4ad91b85-a022-47b9-9872-4601a6b3215b\" (UID: \"4ad91b85-a022-47b9-9872-4601a6b3215b\") "
Oct 31 01:42:56.307538 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:56.305237    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad91b85-a022-47b9-9872-4601a6b3215b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ad91b85-a022-47b9-9872-4601a6b3215b-provider-credentials\") pod \"4ad91b85-a022-47b9-9872-4601a6b3215b\" (UID: \"4ad91b85-a022-47b9-9872-4601a6b3215b\") "
Oct 31 01:42:56.327244 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:56.326261    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad91b85-a022-47b9-9872-4601a6b3215b-plugins-config-volume" pod "4ad91b85-a022-47b9-9872-4601a6b3215b" (UID: "4ad91b85-a022-47b9-9872-4601a6b3215b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:42:56.328182 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:56.328152    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad91b85-a022-47b9-9872-4601a6b3215b-routing-overlay-volume" pod "4ad91b85-a022-47b9-9872-4601a6b3215b" (UID: "4ad91b85-a022-47b9-9872-4601a6b3215b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:42:56.371730 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:56.371662    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad91b85-a022-47b9-9872-4601a6b3215b-provider-credentials" pod "4ad91b85-a022-47b9-9872-4601a6b3215b" (UID: "4ad91b85-a022-47b9-9872-4601a6b3215b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:56.396492 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:56.395493    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad91b85-a022-47b9-9872-4601a6b3215b-kube-api-access-b5sp5" pod "4ad91b85-a022-47b9-9872-4601a6b3215b" (UID: "4ad91b85-a022-47b9-9872-4601a6b3215b"). InnerVolumeSpecName "kube-api-access-b5sp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:56.424472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:56.416569    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:42:56.424472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:56.416654    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c" podUID="ced27998-939e-4734-9904-18c4dc0b1170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:42:56.424472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:56.417904    5650 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ad91b85-a022-47b9-9872-4601a6b3215b-routing-overlay-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:56.424472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:56.417933    5650 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ad91b85-a022-47b9-9872-4601a6b3215b-provider-credentials\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:56.424472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:56.417950    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad91b85-a022-47b9-9872-4601a6b3215b-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:56.424472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:56.417966    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5sp5\" (UniqueName: \"kubernetes.io/projected/4ad91b85-a022-47b9-9872-4601a6b3215b-kube-api-access-b5sp5\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:42:56.467719 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:56.467678    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:42:56.468027 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:56.467990    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h" podUID="f14608d5-81ae-4ba3-8986-a8e602befe12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 01:42:56.469105 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:56.469082    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zznwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:42:56.469267 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:56.469246    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh" podUID="b4ebd16b-3010-4a21-abd5-8b1f7e8c4905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:42:57.062118 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:57.057646    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv" podUID="4ad91b85-a022-47b9-9872-4601a6b3215b" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.68:50052\" within 1s: context deadline exceeded"
Oct 31 01:42:57.062118 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:42:57.058046    5650 logging.go:55] [core] [Channel #314 SubChannel #315]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.68:50052", ServerName: "10.134.0.68:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.68:50052: operation was canceled"
Oct 31 01:42:57.086472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:57.085926    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv"
Oct 31 01:42:57.086472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:57.086029    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv" event={"ID":"4ad91b85-a022-47b9-9872-4601a6b3215b","Type":"ContainerDied","Data":"45aabeb4645b15599906063eb5eb9b41a2f90947ea792911fac81b0985b3afbd"}
Oct 31 01:42:57.086472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:57.086091    5650 scope.go:122] "RemoveContainer" containerID="48b845e6fe4ceea3a802e712d94281997cdf3f6002b9d14f8ca00b788bd2181c"
Oct 31 01:42:57.136481 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:57.135510    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv"]
Oct 31 01:42:57.140474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:57.136923    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d6d4dd5c-xrldv"]
Oct 31 01:42:57.463360 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:57.463318    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:42:57.464204 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:57.464164    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h" podUID="f14608d5-81ae-4ba3-8986-a8e602befe12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 01:42:57.472301 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:57.466535    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:42:57.472301 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:57.471248    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c" podUID="ced27998-939e-4734-9904-18c4dc0b1170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:42:57.528615 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:57.528569    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zznwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:42:57.528907 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:57.528849    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh" podUID="b4ebd16b-3010-4a21-abd5-8b1f7e8c4905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:42:58.216471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:58.196630    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad91b85-a022-47b9-9872-4601a6b3215b" path="/var/lib/kubelet/pods/4ad91b85-a022-47b9-9872-4601a6b3215b/volumes"
Oct 31 01:42:58.448124 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:58.448084    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:42:58.448377 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:58.448344    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h" podUID="f14608d5-81ae-4ba3-8986-a8e602befe12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 01:42:58.519652 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:58.519552    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:42:58.519913 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:58.519881    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c" podUID="ced27998-939e-4734-9904-18c4dc0b1170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:42:58.574472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:58.571647    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zznwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:42:58.574472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:58.571720    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh" podUID="b4ebd16b-3010-4a21-abd5-8b1f7e8c4905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:42:58.630472 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:42:58.622854    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:42:58.702472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:58.701527    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:58.706470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:58.702958    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:58.711469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:58.708923    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jpp7c"
Oct 31 01:42:58.772137 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:58.772058    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:58.773180 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:58.773011    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:58.774208 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:58.774184    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zznwh"
Oct 31 01:42:59.440369 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:59.440333    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:42:59.440789 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:59.440392    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h" podUID="f14608d5-81ae-4ba3-8986-a8e602befe12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 01:42:59.992827 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:42:59.992794    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:43:00.032332 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:00.032300    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:43:00.440317 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:00.440225    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:43:00.440317 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:00.440297    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h" podUID="f14608d5-81ae-4ba3-8986-a8e602befe12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 01:43:01.196986 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.196938    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj"]
Oct 31 01:43:01.200895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.200856    5650 cpu_manager.go:414] "containerMap: removing container" podUID="4ad91b85-a022-47b9-9872-4601a6b3215b" containerName="payload-processing"
Oct 31 01:43:01.200895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.200887    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad91b85-a022-47b9-9872-4601a6b3215b" containerName="payload-processing"
Oct 31 01:43:01.201108 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.201043    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad91b85-a022-47b9-9872-4601a6b3215b" containerName="payload-processing"
Oct 31 01:43:01.214733 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.214698    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj"]
Oct 31 01:43:01.214912 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.214826    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj"
Oct 31 01:43:01.232208 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.230242    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 01:43:01.232208 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.230360    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 01:43:01.235387 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.234756    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 01:43:01.235387 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.234828    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 01:43:01.235387 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.234922    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-e1bb458aa6fe\""
Oct 31 01:43:01.235387 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.235131    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-h627d\""
Oct 31 01:43:01.363314 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.363268    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/90d6b45f-0f92-457f-98c7-9a16c7591a42-provider-credentials\") pod \"payload-processing-external-model-d4c9ccc64-4zhmj\" (UID: \"90d6b45f-0f92-457f-98c7-9a16c7591a42\") " pod="models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj"
Oct 31 01:43:01.363314 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.363317    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90d6b45f-0f92-457f-98c7-9a16c7591a42-plugins-config-volume\") pod \"payload-processing-external-model-d4c9ccc64-4zhmj\" (UID: \"90d6b45f-0f92-457f-98c7-9a16c7591a42\") " pod="models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj"
Oct 31 01:43:01.363556 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.363430    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pmlj\" (UniqueName: \"kubernetes.io/projected/90d6b45f-0f92-457f-98c7-9a16c7591a42-kube-api-access-4pmlj\") pod \"payload-processing-external-model-d4c9ccc64-4zhmj\" (UID: \"90d6b45f-0f92-457f-98c7-9a16c7591a42\") " pod="models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj"
Oct 31 01:43:01.363673 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.363650    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90d6b45f-0f92-457f-98c7-9a16c7591a42-routing-overlay-volume\") pod \"payload-processing-external-model-d4c9ccc64-4zhmj\" (UID: \"90d6b45f-0f92-457f-98c7-9a16c7591a42\") " pod="models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj"
Oct 31 01:43:01.464848 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.464775    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90d6b45f-0f92-457f-98c7-9a16c7591a42-routing-overlay-volume\") pod \"payload-processing-external-model-d4c9ccc64-4zhmj\" (UID: \"90d6b45f-0f92-457f-98c7-9a16c7591a42\") " pod="models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj"
Oct 31 01:43:01.465491 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.464917    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/90d6b45f-0f92-457f-98c7-9a16c7591a42-provider-credentials\") pod \"payload-processing-external-model-d4c9ccc64-4zhmj\" (UID: \"90d6b45f-0f92-457f-98c7-9a16c7591a42\") " pod="models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj"
Oct 31 01:43:01.465491 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.464951    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90d6b45f-0f92-457f-98c7-9a16c7591a42-plugins-config-volume\") pod \"payload-processing-external-model-d4c9ccc64-4zhmj\" (UID: \"90d6b45f-0f92-457f-98c7-9a16c7591a42\") " pod="models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj"
Oct 31 01:43:01.465491 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.464994    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pmlj\" (UniqueName: \"kubernetes.io/projected/90d6b45f-0f92-457f-98c7-9a16c7591a42-kube-api-access-4pmlj\") pod \"payload-processing-external-model-d4c9ccc64-4zhmj\" (UID: \"90d6b45f-0f92-457f-98c7-9a16c7591a42\") " pod="models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj"
Oct 31 01:43:01.467339 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.467311    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90d6b45f-0f92-457f-98c7-9a16c7591a42-routing-overlay-volume\") pod \"payload-processing-external-model-d4c9ccc64-4zhmj\" (UID: \"90d6b45f-0f92-457f-98c7-9a16c7591a42\") " pod="models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj"
Oct 31 01:43:01.468377 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.468349    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90d6b45f-0f92-457f-98c7-9a16c7591a42-plugins-config-volume\") pod \"payload-processing-external-model-d4c9ccc64-4zhmj\" (UID: \"90d6b45f-0f92-457f-98c7-9a16c7591a42\") " pod="models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj"
Oct 31 01:43:01.472768 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.470951    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/90d6b45f-0f92-457f-98c7-9a16c7591a42-provider-credentials\") pod \"payload-processing-external-model-d4c9ccc64-4zhmj\" (UID: \"90d6b45f-0f92-457f-98c7-9a16c7591a42\") " pod="models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj"
Oct 31 01:43:01.480000 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.479972    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pmlj\" (UniqueName: \"kubernetes.io/projected/90d6b45f-0f92-457f-98c7-9a16c7591a42-kube-api-access-4pmlj\") pod \"payload-processing-external-model-d4c9ccc64-4zhmj\" (UID: \"90d6b45f-0f92-457f-98c7-9a16c7591a42\") " pod="models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj"
Oct 31 01:43:01.512367 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.512331    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:43:01.512831 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.512805    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:43:01.513653 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.513635    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 01:43:01.557075 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.557045    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj"
Oct 31 01:43:01.890225 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:01.890190    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj"]
Oct 31 01:43:01.890812 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:43:01.890787    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d6b45f_0f92_457f_98c7_9a16c7591a42.slice/crio-c9f3f8f06f7c584fb003fc6b1f023f36592939114d49b288248c693812ad0da5 WatchSource:0}: Error finding container c9f3f8f06f7c584fb003fc6b1f023f36592939114d49b288248c693812ad0da5: Status 404 returned error can't find the container with id c9f3f8f06f7c584fb003fc6b1f023f36592939114d49b288248c693812ad0da5
Oct 31 01:43:02.150600 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:02.150515    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj" event={"ID":"90d6b45f-0f92-457f-98c7-9a16c7591a42","Type":"ContainerStarted","Data":"aca4f06322a6e75b9fe1e367446a87691f5596bb085b9700657b6b5edb8b7c8d"}
Oct 31 01:43:02.150600 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:02.150571    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj" event={"ID":"90d6b45f-0f92-457f-98c7-9a16c7591a42","Type":"ContainerStarted","Data":"c9f3f8f06f7c584fb003fc6b1f023f36592939114d49b288248c693812ad0da5"}
Oct 31 01:43:02.173275 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:02.173222    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj" podStartSLOduration=1.173206177 podStartE2EDuration="1.173206177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:02.172361028 +0000 UTC m=+1338.176075320" watchObservedRunningTime="2026-10-31 01:43:02.173206177 +0000 UTC m=+1338.176920467"
Oct 31 01:43:03.164432 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:03.164395    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj"
Oct 31 01:43:03.628044 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:03.628006    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"]
Oct 31 01:43:03.631802 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:03.631779    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:03.640914 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:03.640892    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6ae519\"/\"distinct-2-w2-6ae519-kserve-self-signed-certs\""
Oct 31 01:43:03.641894 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:03.641867    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"]
Oct 31 01:43:03.691329 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:03.691290    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqmlr\" (UniqueName: \"kubernetes.io/projected/0cbae488-58ba-454e-874a-8849b71ec03a-kube-api-access-lqmlr\") pod \"distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47\" (UID: \"0cbae488-58ba-454e-874a-8849b71ec03a\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:03.691493 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:03.691359    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-dshm\") pod \"distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47\" (UID: \"0cbae488-58ba-454e-874a-8849b71ec03a\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:03.691563 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:03.691513    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbae488-58ba-454e-874a-8849b71ec03a-tls-certs\") pod \"distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47\" (UID: \"0cbae488-58ba-454e-874a-8849b71ec03a\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:03.691600 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:03.691584    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-tmp-dir\") pod \"distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47\" (UID: \"0cbae488-58ba-454e-874a-8849b71ec03a\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:03.691674 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:03.691651    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-home\") pod \"distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47\" (UID: \"0cbae488-58ba-454e-874a-8849b71ec03a\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:03.691707 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:03.691685    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-model-cache\") pod \"distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47\" (UID: \"0cbae488-58ba-454e-874a-8849b71ec03a\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:03.794932 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:03.793839    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbae488-58ba-454e-874a-8849b71ec03a-tls-certs\") pod \"distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47\" (UID: \"0cbae488-58ba-454e-874a-8849b71ec03a\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:03.794932 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:03.793937    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-tmp-dir\") pod \"distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47\" (UID: \"0cbae488-58ba-454e-874a-8849b71ec03a\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:03.794932 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:03.793970    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-home\") pod \"distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47\" (UID: \"0cbae488-58ba-454e-874a-8849b71ec03a\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:03.794932 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:03.793999    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-model-cache\") pod \"distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47\" (UID: \"0cbae488-58ba-454e-874a-8849b71ec03a\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:03.794932 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:03.794125    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqmlr\" (UniqueName: \"kubernetes.io/projected/0cbae488-58ba-454e-874a-8849b71ec03a-kube-api-access-lqmlr\") pod \"distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47\" (UID: \"0cbae488-58ba-454e-874a-8849b71ec03a\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:03.794932 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:03.794189    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-dshm\") pod \"distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47\" (UID: \"0cbae488-58ba-454e-874a-8849b71ec03a\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:03.795434 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:03.795030    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-home\") pod \"distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47\" (UID: \"0cbae488-58ba-454e-874a-8849b71ec03a\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:03.798832 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:03.798797    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-tmp-dir\") pod \"distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47\" (UID: \"0cbae488-58ba-454e-874a-8849b71ec03a\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:03.799032 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:03.799016    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-model-cache\") pod \"distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47\" (UID: \"0cbae488-58ba-454e-874a-8849b71ec03a\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:03.806732 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:03.806688    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbae488-58ba-454e-874a-8849b71ec03a-tls-certs\") pod \"distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47\" (UID: \"0cbae488-58ba-454e-874a-8849b71ec03a\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:03.817394 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:03.817314    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqmlr\" (UniqueName: \"kubernetes.io/projected/0cbae488-58ba-454e-874a-8849b71ec03a-kube-api-access-lqmlr\") pod \"distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47\" (UID: \"0cbae488-58ba-454e-874a-8849b71ec03a\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:03.827178 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:03.826802    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-dshm\") pod \"distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47\" (UID: \"0cbae488-58ba-454e-874a-8849b71ec03a\") " pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:03.946314 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:03.946216    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:04.235637 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:04.235604    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"]
Oct 31 01:43:04.236997 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:43:04.236969    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cbae488_58ba_454e_874a_8849b71ec03a.slice/crio-2e184427c35997bbc35cf2a69ca7e2093a420598ce24ac3bc2851cef7322eb53 WatchSource:0}: Error finding container 2e184427c35997bbc35cf2a69ca7e2093a420598ce24ac3bc2851cef7322eb53: Status 404 returned error can't find the container with id 2e184427c35997bbc35cf2a69ca7e2093a420598ce24ac3bc2851cef7322eb53
Oct 31 01:43:05.179714 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:05.179679    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" event={"ID":"0cbae488-58ba-454e-874a-8849b71ec03a","Type":"ContainerStarted","Data":"951f29eef385521b041d51dc2ef435ac0dfe4d1f68768accc0e23a317a0d052b"}
Oct 31 01:43:05.179913 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:05.179726    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" event={"ID":"0cbae488-58ba-454e-874a-8849b71ec03a","Type":"ContainerStarted","Data":"2e184427c35997bbc35cf2a69ca7e2093a420598ce24ac3bc2851cef7322eb53"}
Oct 31 01:43:05.208324 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:05.208263    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podStartSLOduration=2.208249776 podStartE2EDuration="2.208249776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:05.206862446 +0000 UTC m=+1341.210576738" watchObservedRunningTime="2026-10-31 01:43:05.208249776 +0000 UTC m=+1341.211964067"
Oct 31 01:43:05.947056 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:05.946999    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:10.103571 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:10.102566    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6k8js container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:43:10.103571 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:10.102655    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js" podUID="b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:10.125511 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:10.124583    5650 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xljzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:43:10.125511 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:10.124655    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xljzj" podUID="a125e272-0ca1-47ca-9f22-e46a649b3f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:11.429779 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.429742    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"]
Oct 31 01:43:11.433991 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.433966    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.448564 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.448532    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"]
Oct 31 01:43:11.476504 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.476469    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2118bab-9cba-42f7-814d-730046ef72a2-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.476693 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.476517    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.476693 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.476591    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ds9z\" (UniqueName: \"kubernetes.io/projected/d2118bab-9cba-42f7-814d-730046ef72a2-kube-api-access-7ds9z\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.476693 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.476628    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.476808 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.476737    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2118bab-9cba-42f7-814d-730046ef72a2-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.476808 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.476756    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2118bab-9cba-42f7-814d-730046ef72a2-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.476883 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.476847    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.476922 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.476908    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.477015 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.476995    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.578495 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.578431    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2118bab-9cba-42f7-814d-730046ef72a2-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.578689 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.578538    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.578689 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.578637    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.578871 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.578706    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.578871 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.578766    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2118bab-9cba-42f7-814d-730046ef72a2-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.578871 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.578808    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.578871 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.578836    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ds9z\" (UniqueName: \"kubernetes.io/projected/d2118bab-9cba-42f7-814d-730046ef72a2-kube-api-access-7ds9z\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.578871 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.578867    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.579120 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.578972    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2118bab-9cba-42f7-814d-730046ef72a2-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.579120 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.579012    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.579231 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.579201    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2118bab-9cba-42f7-814d-730046ef72a2-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.579531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.579505    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.579644 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.579549    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.579706 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.579659    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.581136 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.581101    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.581837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.581815    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2118bab-9cba-42f7-814d-730046ef72a2-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.592912 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.592886    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2118bab-9cba-42f7-814d-730046ef72a2-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.593413 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.593389    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ds9z\" (UniqueName: \"kubernetes.io/projected/d2118bab-9cba-42f7-814d-730046ef72a2-kube-api-access-7ds9z\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:11.752655 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:11.752607    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:12.044826 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:12.044797    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"]
Oct 31 01:43:12.046638 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:43:12.046611    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2118bab_9cba_42f7_814d_730046ef72a2.slice/crio-8a59830824ec5daf23a7942ba91d7fd24c0c4b5b7d220fd0d5ef112d5fd8b290 WatchSource:0}: Error finding container 8a59830824ec5daf23a7942ba91d7fd24c0c4b5b7d220fd0d5ef112d5fd8b290: Status 404 returned error can't find the container with id 8a59830824ec5daf23a7942ba91d7fd24c0c4b5b7d220fd0d5ef112d5fd8b290
Oct 31 01:43:12.049430 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:12.049396    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:43:12.049551 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:12.049485    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:43:12.049551 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:12.049519    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:43:12.339805 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:12.339712    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4" event={"ID":"d2118bab-9cba-42f7-814d-730046ef72a2","Type":"ContainerStarted","Data":"bc9360ecec41e1006c1f672e65ab2c886e14eff66250d4cb2303669173966a91"}
Oct 31 01:43:12.339805 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:12.339781    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4" event={"ID":"d2118bab-9cba-42f7-814d-730046ef72a2","Type":"ContainerStarted","Data":"8a59830824ec5daf23a7942ba91d7fd24c0c4b5b7d220fd0d5ef112d5fd8b290"}
Oct 31 01:43:12.366405 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:12.366339    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4" podStartSLOduration=1.366315365 podStartE2EDuration="1.366315365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:12.365036023 +0000 UTC m=+1348.368750336" watchObservedRunningTime="2026-10-31 01:43:12.366315365 +0000 UTC m=+1348.370029659"
Oct 31 01:43:12.753651 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:12.753611    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:13.754551 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:13.754518    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:43:13.754991 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:13.754572    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4" podUID="d2118bab-9cba-42f7-814d-730046ef72a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 01:43:13.947154 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:13.947115    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:13.959773 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:13.959740    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:14.175485 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:14.175389    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj"
Oct 31 01:43:14.373787 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:14.373757    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:14.753966 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:14.753927    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:43:14.754152 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:14.753983    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4" podUID="d2118bab-9cba-42f7-814d-730046ef72a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 01:43:15.753078 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:15.753041    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:43:15.753723 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:15.753666    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4" podUID="d2118bab-9cba-42f7-814d-730046ef72a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 01:43:16.755472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:16.754934    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:43:16.755472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:16.755019    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4" podUID="d2118bab-9cba-42f7-814d-730046ef72a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 01:43:17.753865 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:17.753828    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:43:17.754072 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:17.753888    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4" podUID="d2118bab-9cba-42f7-814d-730046ef72a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 01:43:18.753500 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:18.753460    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:43:18.753929 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:18.753549    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4" podUID="d2118bab-9cba-42f7-814d-730046ef72a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 01:43:19.849175 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:19.849131    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:19.849680 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:19.849522    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:19.850821 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:19.850795    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:43:26.810820 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:26.810738    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"]
Oct 31 01:43:26.816279 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:26.816255    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:26.827893 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:26.827855    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3cf6f4\"/\"distinct-w0-3cf6f4-kserve-self-signed-certs\""
Oct 31 01:43:26.828360 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:26.828338    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"]
Oct 31 01:43:26.968270 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:26.968229    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/346efd30-40a6-4b13-b733-7e5d3e19e92f-tls-certs\") pod \"distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w\" (UID: \"346efd30-40a6-4b13-b733-7e5d3e19e92f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:26.968509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:26.968308    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xzfn\" (UniqueName: \"kubernetes.io/projected/346efd30-40a6-4b13-b733-7e5d3e19e92f-kube-api-access-4xzfn\") pod \"distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w\" (UID: \"346efd30-40a6-4b13-b733-7e5d3e19e92f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:26.968509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:26.968363    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-tmp-dir\") pod \"distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w\" (UID: \"346efd30-40a6-4b13-b733-7e5d3e19e92f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:26.968509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:26.968483    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-model-cache\") pod \"distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w\" (UID: \"346efd30-40a6-4b13-b733-7e5d3e19e92f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:26.968748 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:26.968678    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-dshm\") pod \"distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w\" (UID: \"346efd30-40a6-4b13-b733-7e5d3e19e92f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:26.968748 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:26.968743    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-home\") pod \"distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w\" (UID: \"346efd30-40a6-4b13-b733-7e5d3e19e92f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:27.070352 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:27.070245    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/346efd30-40a6-4b13-b733-7e5d3e19e92f-tls-certs\") pod \"distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w\" (UID: \"346efd30-40a6-4b13-b733-7e5d3e19e92f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:27.070352 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:27.070302    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xzfn\" (UniqueName: \"kubernetes.io/projected/346efd30-40a6-4b13-b733-7e5d3e19e92f-kube-api-access-4xzfn\") pod \"distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w\" (UID: \"346efd30-40a6-4b13-b733-7e5d3e19e92f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:27.070352 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:27.070344    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-tmp-dir\") pod \"distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w\" (UID: \"346efd30-40a6-4b13-b733-7e5d3e19e92f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:27.070694 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:27.070424    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-model-cache\") pod \"distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w\" (UID: \"346efd30-40a6-4b13-b733-7e5d3e19e92f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:27.070694 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:27.070495    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-dshm\") pod \"distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w\" (UID: \"346efd30-40a6-4b13-b733-7e5d3e19e92f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:27.070694 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:27.070549    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-home\") pod \"distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w\" (UID: \"346efd30-40a6-4b13-b733-7e5d3e19e92f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:27.070960 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:27.070928    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-model-cache\") pod \"distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w\" (UID: \"346efd30-40a6-4b13-b733-7e5d3e19e92f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:27.070960 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:27.070954    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-home\") pod \"distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w\" (UID: \"346efd30-40a6-4b13-b733-7e5d3e19e92f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:27.071124 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:27.071002    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-tmp-dir\") pod \"distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w\" (UID: \"346efd30-40a6-4b13-b733-7e5d3e19e92f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:27.072973 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:27.072943    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-dshm\") pod \"distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w\" (UID: \"346efd30-40a6-4b13-b733-7e5d3e19e92f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:27.073074 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:27.073063    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/346efd30-40a6-4b13-b733-7e5d3e19e92f-tls-certs\") pod \"distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w\" (UID: \"346efd30-40a6-4b13-b733-7e5d3e19e92f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:27.083952 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:27.083925    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xzfn\" (UniqueName: \"kubernetes.io/projected/346efd30-40a6-4b13-b733-7e5d3e19e92f-kube-api-access-4xzfn\") pod \"distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w\" (UID: \"346efd30-40a6-4b13-b733-7e5d3e19e92f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:27.131377 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:27.131332    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:27.420744 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:27.420713    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"]
Oct 31 01:43:27.421564 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:43:27.421532    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod346efd30_40a6_4b13_b733_7e5d3e19e92f.slice/crio-d6d28560c0dfc033b64926982ed2ba2bba48d5a89fa6ff700504df54f69350d4 WatchSource:0}: Error finding container d6d28560c0dfc033b64926982ed2ba2bba48d5a89fa6ff700504df54f69350d4: Status 404 returned error can't find the container with id d6d28560c0dfc033b64926982ed2ba2bba48d5a89fa6ff700504df54f69350d4
Oct 31 01:43:27.484672 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:27.484637    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" event={"ID":"346efd30-40a6-4b13-b733-7e5d3e19e92f","Type":"ContainerStarted","Data":"d6d28560c0dfc033b64926982ed2ba2bba48d5a89fa6ff700504df54f69350d4"}
Oct 31 01:43:28.492950 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:28.492914    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" event={"ID":"346efd30-40a6-4b13-b733-7e5d3e19e92f","Type":"ContainerStarted","Data":"a0b4763bc801b304d13780d4d24919f48a1a48b1eab6e241afdb1de598fa931f"}
Oct 31 01:43:28.516310 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:28.516259    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podStartSLOduration=2.516245097 podStartE2EDuration="2.516245097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:28.514708208 +0000 UTC m=+1364.518422500" watchObservedRunningTime="2026-10-31 01:43:28.516245097 +0000 UTC m=+1364.519959716"
Oct 31 01:43:28.553160 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:43:28.553115    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:43:29.131972 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:29.131940    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:34.185642 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:34.185606    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj"]
Oct 31 01:43:37.131537 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:37.131491    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:37.144906 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:37.144842    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:37.573166 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:37.573136    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:38.209891 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.209853    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"]
Oct 31 01:43:38.224106 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.224073    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.230105 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.230071    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"]
Oct 31 01:43:38.313206 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.313167    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.313376 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.313231    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.313376 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.313318    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.313471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.313393    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.313471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.313426    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.313543 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.313472    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.313576 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.313559    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.313609 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.313598    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.313712 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.313699    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmg97\" (UniqueName: \"kubernetes.io/projected/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-kube-api-access-cmg97\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.414730 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.414693    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.414933 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.414788    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmg97\" (UniqueName: \"kubernetes.io/projected/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-kube-api-access-cmg97\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.414933 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.414836    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.414933 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.414884    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.415100 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.414931    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.415100 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.414975    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.415100 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.415022    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.415100 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.415064    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.415309 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.415150    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.415309 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.415170    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.415419 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.415402    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.415537 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.415514    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.415690 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.415663    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.415847 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.415667    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.417640 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.417615    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.417748 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.417713    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.433375 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.432817    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.435216 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.435187    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmg97\" (UniqueName: \"kubernetes.io/projected/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-kube-api-access-cmg97\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.545595 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.545493    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:38.833331 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.833299    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"]
Oct 31 01:43:38.833972 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:43:38.833932    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddedb244a_f01c_463e_8201_c6d0ad8e3f3f.slice/crio-aba6fbb2f54c7f5414a9318003a49e72d1d845deba71a8a330953315b765254a WatchSource:0}: Error finding container aba6fbb2f54c7f5414a9318003a49e72d1d845deba71a8a330953315b765254a: Status 404 returned error can't find the container with id aba6fbb2f54c7f5414a9318003a49e72d1d845deba71a8a330953315b765254a
Oct 31 01:43:38.836483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.836436    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:43:38.836572 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.836518    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:43:38.836572 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:38.836546    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:43:39.186855 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.186763    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj" podUID="90d6b45f-0f92-457f-98c7-9a16c7591a42" containerName="payload-processing" containerID="cri-o://aca4f06322a6e75b9fe1e367446a87691f5596bb085b9700657b6b5edb8b7c8d" gracePeriod=25
Oct 31 01:43:39.484676 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.484644    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj"
Oct 31 01:43:39.534162 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.534121    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90d6b45f-0f92-457f-98c7-9a16c7591a42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90d6b45f-0f92-457f-98c7-9a16c7591a42-plugins-config-volume\") pod \"90d6b45f-0f92-457f-98c7-9a16c7591a42\" (UID: \"90d6b45f-0f92-457f-98c7-9a16c7591a42\") "
Oct 31 01:43:39.536752 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.535648    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90d6b45f-0f92-457f-98c7-9a16c7591a42-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90d6b45f-0f92-457f-98c7-9a16c7591a42-routing-overlay-volume\") pod \"90d6b45f-0f92-457f-98c7-9a16c7591a42\" (UID: \"90d6b45f-0f92-457f-98c7-9a16c7591a42\") "
Oct 31 01:43:39.536752 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.536575    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90d6b45f-0f92-457f-98c7-9a16c7591a42-provider-credentials\" (UniqueName: \"kubernetes.io/projected/90d6b45f-0f92-457f-98c7-9a16c7591a42-provider-credentials\") pod \"90d6b45f-0f92-457f-98c7-9a16c7591a42\" (UID: \"90d6b45f-0f92-457f-98c7-9a16c7591a42\") "
Oct 31 01:43:39.536752 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.536630    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90d6b45f-0f92-457f-98c7-9a16c7591a42-kube-api-access-4pmlj\" (UniqueName: \"kubernetes.io/projected/90d6b45f-0f92-457f-98c7-9a16c7591a42-kube-api-access-4pmlj\") pod \"90d6b45f-0f92-457f-98c7-9a16c7591a42\" (UID: \"90d6b45f-0f92-457f-98c7-9a16c7591a42\") "
Oct 31 01:43:39.538885 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.535988    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90d6b45f-0f92-457f-98c7-9a16c7591a42-plugins-config-volume" pod "90d6b45f-0f92-457f-98c7-9a16c7591a42" (UID: "90d6b45f-0f92-457f-98c7-9a16c7591a42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:39.538885 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.536353    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90d6b45f-0f92-457f-98c7-9a16c7591a42-routing-overlay-volume" pod "90d6b45f-0f92-457f-98c7-9a16c7591a42" (UID: "90d6b45f-0f92-457f-98c7-9a16c7591a42"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:39.546615 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.546436    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d6b45f-0f92-457f-98c7-9a16c7591a42-provider-credentials" pod "90d6b45f-0f92-457f-98c7-9a16c7591a42" (UID: "90d6b45f-0f92-457f-98c7-9a16c7591a42"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:39.546615 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.546581    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d6b45f-0f92-457f-98c7-9a16c7591a42-kube-api-access-4pmlj" pod "90d6b45f-0f92-457f-98c7-9a16c7591a42" (UID: "90d6b45f-0f92-457f-98c7-9a16c7591a42"). InnerVolumeSpecName "kube-api-access-4pmlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:39.580808 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.580775    5650 generic.go:356] "Generic (PLEG): container finished" podID="90d6b45f-0f92-457f-98c7-9a16c7591a42" containerID="aca4f06322a6e75b9fe1e367446a87691f5596bb085b9700657b6b5edb8b7c8d" exitCode=0
Oct 31 01:43:39.581061 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.580905    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj"
Oct 31 01:43:39.581237 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.580939    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj" event={"ID":"90d6b45f-0f92-457f-98c7-9a16c7591a42","Type":"ContainerDied","Data":"aca4f06322a6e75b9fe1e367446a87691f5596bb085b9700657b6b5edb8b7c8d"}
Oct 31 01:43:39.581306 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.581274    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj" event={"ID":"90d6b45f-0f92-457f-98c7-9a16c7591a42","Type":"ContainerDied","Data":"c9f3f8f06f7c584fb003fc6b1f023f36592939114d49b288248c693812ad0da5"}
Oct 31 01:43:39.581357 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.581311    5650 scope.go:122] "RemoveContainer" containerID="aca4f06322a6e75b9fe1e367446a87691f5596bb085b9700657b6b5edb8b7c8d"
Oct 31 01:43:39.583039 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.583008    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7" event={"ID":"dedb244a-f01c-463e-8201-c6d0ad8e3f3f","Type":"ContainerStarted","Data":"e6a55bdbddfd6bf37a277e04793762730a4c46f074084239cea74340edd8107b"}
Oct 31 01:43:39.583140 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.583059    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7" event={"ID":"dedb244a-f01c-463e-8201-c6d0ad8e3f3f","Type":"ContainerStarted","Data":"aba6fbb2f54c7f5414a9318003a49e72d1d845deba71a8a330953315b765254a"}
Oct 31 01:43:39.592977 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.592949    5650 scope.go:122] "RemoveContainer" containerID="aca4f06322a6e75b9fe1e367446a87691f5596bb085b9700657b6b5edb8b7c8d"
Oct 31 01:43:39.593292 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:43:39.593269    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aca4f06322a6e75b9fe1e367446a87691f5596bb085b9700657b6b5edb8b7c8d\": container with ID starting with aca4f06322a6e75b9fe1e367446a87691f5596bb085b9700657b6b5edb8b7c8d not found: ID does not exist" containerID="aca4f06322a6e75b9fe1e367446a87691f5596bb085b9700657b6b5edb8b7c8d"
Oct 31 01:43:39.593344 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.593300    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aca4f06322a6e75b9fe1e367446a87691f5596bb085b9700657b6b5edb8b7c8d"} err="failed to get container status \"aca4f06322a6e75b9fe1e367446a87691f5596bb085b9700657b6b5edb8b7c8d\": rpc error: code = NotFound desc = could not find container \"aca4f06322a6e75b9fe1e367446a87691f5596bb085b9700657b6b5edb8b7c8d\": container with ID starting with aca4f06322a6e75b9fe1e367446a87691f5596bb085b9700657b6b5edb8b7c8d not found: ID does not exist"
Oct 31 01:43:39.629015 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.628834    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7" podStartSLOduration=1.628813157 podStartE2EDuration="1.628813157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:39.62305016 +0000 UTC m=+1375.626764453" watchObservedRunningTime="2026-10-31 01:43:39.628813157 +0000 UTC m=+1375.632527448"
Oct 31 01:43:39.639018 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.638991    5650 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/90d6b45f-0f92-457f-98c7-9a16c7591a42-provider-credentials\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:43:39.639018 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.639014    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pmlj\" (UniqueName: \"kubernetes.io/projected/90d6b45f-0f92-457f-98c7-9a16c7591a42-kube-api-access-4pmlj\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:43:39.639210 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.639027    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90d6b45f-0f92-457f-98c7-9a16c7591a42-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:43:39.639210 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.639043    5650 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90d6b45f-0f92-457f-98c7-9a16c7591a42-routing-overlay-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:43:39.643301 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.643276    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj"]
Oct 31 01:43:39.648094 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:39.648066    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-d4c9ccc64-4zhmj"]
Oct 31 01:43:40.129941 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:40.129903    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90d6b45f-0f92-457f-98c7-9a16c7591a42" path="/var/lib/kubelet/pods/90d6b45f-0f92-457f-98c7-9a16c7591a42/volumes"
Oct 31 01:43:40.546180 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:40.546144    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:40.547547 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:40.547515    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 01:43:40.547615 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:40.547576    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7" podUID="dedb244a-f01c-463e-8201-c6d0ad8e3f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 01:43:41.545975 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:41.545938    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 01:43:41.546158 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:41.545999    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7" podUID="dedb244a-f01c-463e-8201-c6d0ad8e3f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 01:43:42.546089 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:42.546051    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 01:43:42.546563 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:42.546119    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7" podUID="dedb244a-f01c-463e-8201-c6d0ad8e3f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 01:43:43.546150 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:43.546113    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 01:43:43.546599 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:43.546172    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7" podUID="dedb244a-f01c-463e-8201-c6d0ad8e3f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 01:43:44.546044 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:44.546002    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 01:43:44.546217 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:44.546060    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7" podUID="dedb244a-f01c-463e-8201-c6d0ad8e3f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 01:43:45.548898 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:45.548856    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 01:43:45.550000 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:45.549558    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7" podUID="dedb244a-f01c-463e-8201-c6d0ad8e3f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 01:43:47.368470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:47.362794    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 01:43:47.427056 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:47.425606    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:47.490471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:47.489593    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:47.557470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:47.556578    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:43:47.557470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:47.556663    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7" podUID="dedb244a-f01c-463e-8201-c6d0ad8e3f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:47.558703 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:47.558657    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:47.567088 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:47.566996    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 31 01:43:47.999646 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:47.999572    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 01:43:48.148483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:48.143273    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded"
Oct 31 01:43:48.413741 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:48.410421    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:48.433987 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:48.428565    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:48.433987 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:48.429335    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:43:48.493048 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:48.493002    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:48.493944 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:48.493917    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:43:48.562418 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:48.562371    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:48.601473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:48.567339    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:43:48.777237 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:48.602586    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 31 01:43:48.777601 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:48.653576    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:43:48.777780 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:48.777748    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7" podUID="dedb244a-f01c-463e-8201-c6d0ad8e3f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:48.777914 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:48.755094    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded"
Oct 31 01:43:48.781407 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:48.781365    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:49.049534 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:49.024320    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 01:43:49.121657 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:49.098503    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:43:49.368919 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:49.368807    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:49.432269 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:49.432226    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:49.433270 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:49.433242    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:43:49.439578 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:49.439537    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:49.506880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:49.499623    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:49.574870 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:49.574647    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 01:43:49.810585 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:49.796386    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:49.836504 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:49.827926    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:43:49.836504 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:49.827990    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7" podUID="dedb244a-f01c-463e-8201-c6d0ad8e3f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:49.981473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:49.978505    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"]
Oct 31 01:43:49.981473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:49.979340    5650 cpu_manager.go:414] "containerMap: removing container" podUID="90d6b45f-0f92-457f-98c7-9a16c7591a42" containerName="payload-processing"
Oct 31 01:43:49.981473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:49.979365    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90d6b45f-0f92-457f-98c7-9a16c7591a42" containerName="payload-processing"
Oct 31 01:43:49.981473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:49.979611    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="90d6b45f-0f92-457f-98c7-9a16c7591a42" containerName="payload-processing"
Oct 31 01:43:50.121554 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.115520    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:50.166492 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.154868    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:43:50.291055 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.291005    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-tmp-dir\") pod \"distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn\" (UID: \"0c26cdd6-a863-4201-a1e9-f38153be62fc\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:43:50.325641 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.325604    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c26cdd6-a863-4201-a1e9-f38153be62fc-tls-certs\") pod \"distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn\" (UID: \"0c26cdd6-a863-4201-a1e9-f38153be62fc\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:43:50.325968 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.325943    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwrfc\" (UniqueName: \"kubernetes.io/projected/0c26cdd6-a863-4201-a1e9-f38153be62fc-kube-api-access-mwrfc\") pod \"distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn\" (UID: \"0c26cdd6-a863-4201-a1e9-f38153be62fc\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:43:50.326205 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.326187    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-home\") pod \"distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn\" (UID: \"0c26cdd6-a863-4201-a1e9-f38153be62fc\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:43:50.326334 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.326317    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-model-cache\") pod \"distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn\" (UID: \"0c26cdd6-a863-4201-a1e9-f38153be62fc\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:43:50.332689 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.332656    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-dshm\") pod \"distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn\" (UID: \"0c26cdd6-a863-4201-a1e9-f38153be62fc\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:43:50.349497 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.315498    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3cf6f4\"/\"distinct-2-w0-3cf6f4-kserve-self-signed-certs\""
Oct 31 01:43:50.460246 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.442611    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:50.629473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.496152    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 01:43:50.650971 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.504958    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:50.651778 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.505001    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:50.652198 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.505708    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c26cdd6-a863-4201-a1e9-f38153be62fc-tls-certs\") pod \"distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn\" (UID: \"0c26cdd6-a863-4201-a1e9-f38153be62fc\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:43:50.652438 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.652415    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwrfc\" (UniqueName: \"kubernetes.io/projected/0c26cdd6-a863-4201-a1e9-f38153be62fc-kube-api-access-mwrfc\") pod \"distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn\" (UID: \"0c26cdd6-a863-4201-a1e9-f38153be62fc\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:43:50.652713 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.652692    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-home\") pod \"distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn\" (UID: \"0c26cdd6-a863-4201-a1e9-f38153be62fc\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:43:50.652847 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.652831    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-model-cache\") pod \"distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn\" (UID: \"0c26cdd6-a863-4201-a1e9-f38153be62fc\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:43:50.652965 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.652948    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-dshm\") pod \"distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn\" (UID: \"0c26cdd6-a863-4201-a1e9-f38153be62fc\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:43:50.653244 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.653226    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-tmp-dir\") pod \"distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn\" (UID: \"0c26cdd6-a863-4201-a1e9-f38153be62fc\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:43:50.653851 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.653828    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-tmp-dir\") pod \"distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn\" (UID: \"0c26cdd6-a863-4201-a1e9-f38153be62fc\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:43:50.654003 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.572106    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c26cdd6-a863-4201-a1e9-f38153be62fc-tls-certs\") pod \"distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn\" (UID: \"0c26cdd6-a863-4201-a1e9-f38153be62fc\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:43:50.657772 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.657747    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-home\") pod \"distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn\" (UID: \"0c26cdd6-a863-4201-a1e9-f38153be62fc\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:43:50.684021 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.658727    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-model-cache\") pod \"distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn\" (UID: \"0c26cdd6-a863-4201-a1e9-f38153be62fc\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:43:50.814035 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.813906    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:50.842943 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.842890    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:50.868489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.851059    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"]
Oct 31 01:43:50.868489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.851158    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:50.884725 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.884686    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwrfc\" (UniqueName: \"kubernetes.io/projected/0c26cdd6-a863-4201-a1e9-f38153be62fc-kube-api-access-mwrfc\") pod \"distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn\" (UID: \"0c26cdd6-a863-4201-a1e9-f38153be62fc\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:43:50.885025 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.884989    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:43:50.885176 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.885054    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7" podUID="dedb244a-f01c-463e-8201-c6d0ad8e3f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:50.929413 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.919142    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-dshm\") pod \"distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn\" (UID: \"0c26cdd6-a863-4201-a1e9-f38153be62fc\") " pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:43:51.018510 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.997791    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:51.018510 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:50.999255    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:51.130536 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:51.084922    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": context deadline exceeded"
Oct 31 01:43:51.155535 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:51.127057    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:43:51.257696 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:51.257644    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 01:43:51.423535 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:51.420592    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:43:51.479472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:51.458305    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:43:51.481298 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:51.481243    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:51.518623 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:51.518540    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:51.602734 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:51.602703    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:43:51.817469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:51.816402    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:43:51.817469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:51.816517    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:43:51.818506 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:51.818441    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:51.844592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:51.844541    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:51.900620 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:51.897620    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:43:51.933801 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:51.933766    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:43:52.204937 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:52.204809    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"]
Oct 31 01:43:52.207620 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:43:52.207587    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c26cdd6_a863_4201_a1e9_f38153be62fc.slice/crio-ac2c7c9f9ae61fb2d73ba5700c1ad4003680c389121465f0eced134a6d908165 WatchSource:0}: Error finding container ac2c7c9f9ae61fb2d73ba5700c1ad4003680c389121465f0eced134a6d908165: Status 404 returned error can't find the container with id ac2c7c9f9ae61fb2d73ba5700c1ad4003680c389121465f0eced134a6d908165
Oct 31 01:43:52.909083 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:52.909042    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" event={"ID":"0c26cdd6-a863-4201-a1e9-f38153be62fc","Type":"ContainerStarted","Data":"aef5ec36661ebf6466c8a8db3f616c02506fc750511d60548f92d9d8c6a9fec2"}
Oct 31 01:43:52.909534 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:52.909094    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" event={"ID":"0c26cdd6-a863-4201-a1e9-f38153be62fc","Type":"ContainerStarted","Data":"ac2c7c9f9ae61fb2d73ba5700c1ad4003680c389121465f0eced134a6d908165"}
Oct 31 01:43:52.931881 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:52.931828    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podStartSLOduration=3.931813054 podStartE2EDuration="3.931813054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:52.930301159 +0000 UTC m=+1388.934015450" watchObservedRunningTime="2026-10-31 01:43:52.931813054 +0000 UTC m=+1388.935527344"
Oct 31 01:43:53.130405 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:53.130362    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:43:56.577343 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.577264    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"]
Oct 31 01:43:56.583937 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.583909    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"]
Oct 31 01:43:56.586175 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.584237    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.602066 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.602026    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.606164 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.606129    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.606413 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.606392    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.606634 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.606614    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.607340 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.606843    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.607340 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.606904    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.607340 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.606948    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2tt8\" (UniqueName: \"kubernetes.io/projected/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-kube-api-access-d2tt8\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.607340 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.606994    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.607340 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.607069    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.627743 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.627713    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"]
Oct 31 01:43:56.628090 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.628074    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"]
Oct 31 01:43:56.628243 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.628018    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.708239 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.708198    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da0a52af-42a1-45d5-ab84-6696241d5489-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.708430 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.708268    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.708430 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.708309    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.708430 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.708345    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2tt8\" (UniqueName: \"kubernetes.io/projected/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-kube-api-access-d2tt8\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.708430 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.708369    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.708430 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.708411    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.708749 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.708470    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.708749 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.708516    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.708749 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.708586    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da0a52af-42a1-45d5-ab84-6696241d5489-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.708749 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.708680    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.708749 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.708711    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.708994 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.708758    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.708994 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.708820    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.708994 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.708888    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da0a52af-42a1-45d5-ab84-6696241d5489-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.708994 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.708942    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.708994 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.708971    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.709235 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.708998    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zx96\" (UniqueName: \"kubernetes.io/projected/da0a52af-42a1-45d5-ab84-6696241d5489-kube-api-access-4zx96\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.709235 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.709035    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.710166 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.710131    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.710393 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.710372    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.710664 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.710641    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.711173 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.711149    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.711834 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.711767    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.713322 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.713297    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.713718 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.713697    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.722958 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.722911    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.723307 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.723283    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2tt8\" (UniqueName: \"kubernetes.io/projected/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-kube-api-access-d2tt8\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.810200 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.810147    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.810986 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.810961    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da0a52af-42a1-45d5-ab84-6696241d5489-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.811227 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.811207    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.811508 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.811487    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da0a52af-42a1-45d5-ab84-6696241d5489-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.811689 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.811661    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.811804 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.811786    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zx96\" (UniqueName: \"kubernetes.io/projected/da0a52af-42a1-45d5-ab84-6696241d5489-kube-api-access-4zx96\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.811934 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.811918    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.812066 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.812049    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da0a52af-42a1-45d5-ab84-6696241d5489-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.812271 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.812250    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.812841 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.812812    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.813405 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.813380    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.813631 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.813614    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.814595 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.814436    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da0a52af-42a1-45d5-ab84-6696241d5489-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.814595 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.812826    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.819699 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.819671    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.827576 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.827471    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zx96\" (UniqueName: \"kubernetes.io/projected/da0a52af-42a1-45d5-ab84-6696241d5489-kube-api-access-4zx96\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.830077 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.829838    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da0a52af-42a1-45d5-ab84-6696241d5489-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.847471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.837165    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da0a52af-42a1-45d5-ab84-6696241d5489-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:56.911080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.911036    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:56.957468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:56.956848    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:57.814489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:57.810889    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:43:57.814489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:57.811000    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:43:57.814489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:57.811046    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:43:57.881036 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:57.880919    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"]
Oct 31 01:43:57.978372 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:57.978269    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx" event={"ID":"ebc90b15-cef7-4483-a8bb-1a33950f9fa6","Type":"ContainerStarted","Data":"32fbbb9f634534f3892f0aac9e883e3a0b82e477be6541d29002f8a0e3c9f0bc"}
Oct 31 01:43:58.444539 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:58.425120    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 01:43:58.522827 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:58.522768    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"]
Oct 31 01:43:58.586646 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:58.575596    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:58.587694 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:43:58.587107    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0a52af_42a1_45d5_ab84_6696241d5489.slice/crio-7b83ed2eb5e7940f583b3c87663b3d35239ed955044d73f46623060dc912652c WatchSource:0}: Error finding container 7b83ed2eb5e7940f583b3c87663b3d35239ed955044d73f46623060dc912652c: Status 404 returned error can't find the container with id 7b83ed2eb5e7940f583b3c87663b3d35239ed955044d73f46623060dc912652c
Oct 31 01:43:58.625634 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:58.625535    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:43:58.627085 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:58.625894    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:43:58.627085 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:58.625956    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:43:58.702804 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:43:58.699780    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:43:58.743711 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:58.743662    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:43:58.990488 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:58.990369    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8" event={"ID":"da0a52af-42a1-45d5-ab84-6696241d5489","Type":"ContainerStarted","Data":"2b5a02e7e7c0411d921fb61540a74f4fbf32fca53482c1b072d1ec48b6b90004"}
Oct 31 01:43:58.990488 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:58.990457    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8" event={"ID":"da0a52af-42a1-45d5-ab84-6696241d5489","Type":"ContainerStarted","Data":"7b83ed2eb5e7940f583b3c87663b3d35239ed955044d73f46623060dc912652c"}
Oct 31 01:43:58.993589 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:58.993557    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx" event={"ID":"ebc90b15-cef7-4483-a8bb-1a33950f9fa6","Type":"ContainerStarted","Data":"d825246430999b2486f70322c978e1ee5d7b23a345cc29a5910929760122a431"}
Oct 31 01:43:59.046129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:59.046064    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8" podStartSLOduration=3.046044625 podStartE2EDuration="3.046044625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:59.019656991 +0000 UTC m=+1395.023371288" watchObservedRunningTime="2026-10-31 01:43:59.046044625 +0000 UTC m=+1395.049758917"
Oct 31 01:43:59.920078 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:59.920044    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:43:59.920473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:59.920430    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:43:59.920647 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:59.920620    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx" podUID="ebc90b15-cef7-4483-a8bb-1a33950f9fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 01:43:59.957848 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:59.957807    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:43:59.958925 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:59.958896    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 01:43:59.959045 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:43:59.958944    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8" podUID="da0a52af-42a1-45d5-ab84-6696241d5489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 01:44:00.921470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:00.913848    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:44:00.921470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:00.913911    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx" podUID="ebc90b15-cef7-4483-a8bb-1a33950f9fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 01:44:00.958360 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:00.958327    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 01:44:00.958547 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:00.958379    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8" podUID="da0a52af-42a1-45d5-ab84-6696241d5489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 01:44:01.142470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:01.141680    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:44:01.162484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:01.161993    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:44:01.187053 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:01.186943    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx" podStartSLOduration=5.186928983 podStartE2EDuration="5.186928983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:59.049338324 +0000 UTC m=+1395.053052616" watchObservedRunningTime="2026-10-31 01:44:01.186928983 +0000 UTC m=+1397.190643274"
Oct 31 01:44:01.923473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:01.915118    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:44:01.923473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:01.915186    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx" podUID="ebc90b15-cef7-4483-a8bb-1a33950f9fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 01:44:01.958025 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:01.957987    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 01:44:01.958209 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:01.958044    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8" podUID="da0a52af-42a1-45d5-ab84-6696241d5489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 01:44:02.075184 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:02.075153    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:44:02.912470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:02.911917    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:44:02.912470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:02.911997    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx" podUID="ebc90b15-cef7-4483-a8bb-1a33950f9fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 01:44:02.958124 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:02.958082    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 01:44:02.958576 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:02.958150    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8" podUID="da0a52af-42a1-45d5-ab84-6696241d5489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 01:44:03.358578 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.358531    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"]
Oct 31 01:44:03.364691 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.364661    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"]
Oct 31 01:44:03.364872 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.364776    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:03.375597 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.375555    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-be6935\"/\"kube-root-ca.crt\""
Oct 31 01:44:03.375742 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.375640    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-be6935\"/\"openshift-service-ca.crt\""
Oct 31 01:44:03.377436 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.377410    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-be6935\"/\"default-dockercfg-gn5q2\""
Oct 31 01:44:03.377568 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.377508    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-be6935\"/\"distinct-2-w4-be6935-kserve-self-signed-certs\""
Oct 31 01:44:03.463582 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.463527    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-home\") pod \"distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8\" (UID: \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:03.463783 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.463655    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2feec49-d795-4179-8e7b-cbb631a9b5e8-tls-certs\") pod \"distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8\" (UID: \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:03.463783 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.463705    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mddc\" (UniqueName: \"kubernetes.io/projected/a2feec49-d795-4179-8e7b-cbb631a9b5e8-kube-api-access-7mddc\") pod \"distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8\" (UID: \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:03.463907 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.463797    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-tmp-dir\") pod \"distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8\" (UID: \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:03.463967 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.463915    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-dshm\") pod \"distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8\" (UID: \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:03.464018 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.463992    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-model-cache\") pod \"distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8\" (UID: \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:03.570469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.569326    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2feec49-d795-4179-8e7b-cbb631a9b5e8-tls-certs\") pod \"distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8\" (UID: \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:03.570469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.569410    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mddc\" (UniqueName: \"kubernetes.io/projected/a2feec49-d795-4179-8e7b-cbb631a9b5e8-kube-api-access-7mddc\") pod \"distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8\" (UID: \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:03.570469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.569481    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-tmp-dir\") pod \"distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8\" (UID: \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:03.570469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.569594    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-dshm\") pod \"distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8\" (UID: \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:03.570469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.569666    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-model-cache\") pod \"distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8\" (UID: \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:03.570469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.569775    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-home\") pod \"distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8\" (UID: \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:03.570469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.570275    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-home\") pod \"distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8\" (UID: \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:03.585473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.580912    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-tmp-dir\") pod \"distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8\" (UID: \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:03.586714 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.586047    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-model-cache\") pod \"distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8\" (UID: \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:03.587342 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.587301    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-dshm\") pod \"distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8\" (UID: \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:03.616548 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.613050    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2feec49-d795-4179-8e7b-cbb631a9b5e8-tls-certs\") pod \"distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8\" (UID: \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:03.631668 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.631579    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mddc\" (UniqueName: \"kubernetes.io/projected/a2feec49-d795-4179-8e7b-cbb631a9b5e8-kube-api-access-7mddc\") pod \"distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8\" (UID: \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\") " pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:03.680204 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.680171    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:03.923917 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.923817    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:44:03.924220 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.924180    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx" podUID="ebc90b15-cef7-4483-a8bb-1a33950f9fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 01:44:03.958087 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.958045    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 01:44:03.958311 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:03.958292    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8" podUID="da0a52af-42a1-45d5-ab84-6696241d5489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 01:44:04.052704 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.052406    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"]
Oct 31 01:44:04.055581 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:44:04.055535    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2feec49_d795_4179_8e7b_cbb631a9b5e8.slice/crio-a0e371d133f81805f5aeb71b91a9ac8dec8940da9ce172af1e0afbc128647747 WatchSource:0}: Error finding container a0e371d133f81805f5aeb71b91a9ac8dec8940da9ce172af1e0afbc128647747: Status 404 returned error can't find the container with id a0e371d133f81805f5aeb71b91a9ac8dec8940da9ce172af1e0afbc128647747
Oct 31 01:44:04.081579 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.081550    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" event={"ID":"a2feec49-d795-4179-8e7b-cbb631a9b5e8","Type":"ContainerStarted","Data":"a0e371d133f81805f5aeb71b91a9ac8dec8940da9ce172af1e0afbc128647747"}
Oct 31 01:44:04.854467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.853814    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"]
Oct 31 01:44:04.860153 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.860123    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:04.876278 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.876242    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"]
Oct 31 01:44:04.886831 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.886798    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98cd19e-f8d9-4e07-b319-be194ea8f935-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:04.887150 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.887115    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:04.887259 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.887229    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:04.887313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.887296    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:04.887457 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.887416    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-788n8\" (UniqueName: \"kubernetes.io/projected/d98cd19e-f8d9-4e07-b319-be194ea8f935-kube-api-access-788n8\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:04.887540 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.887485    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:04.887540 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.887531    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:04.888061 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.887566    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:04.888061 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.887642    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:04.931636 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.931594    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:44:04.931825 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.931650    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx" podUID="ebc90b15-cef7-4483-a8bb-1a33950f9fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 01:44:04.957488 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.957433    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 01:44:04.957654 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.957504    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8" podUID="da0a52af-42a1-45d5-ab84-6696241d5489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 01:44:04.989142 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.989092    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:04.989663 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.989166    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-788n8\" (UniqueName: \"kubernetes.io/projected/d98cd19e-f8d9-4e07-b319-be194ea8f935-kube-api-access-788n8\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:04.989663 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.989198    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:04.989663 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.989238    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:04.989663 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.989272    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:04.989663 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.989348    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:04.989663 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.989421    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98cd19e-f8d9-4e07-b319-be194ea8f935-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:04.989663 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.989549    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:04.989663 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.989605    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:04.989663 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.989649    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:04.990143 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.989847    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:04.990143 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.990036    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:04.990143 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.990069    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:04.990370 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.990269    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98cd19e-f8d9-4e07-b319-be194ea8f935-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:04.992116 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.992083    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:04.992254 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:04.992203    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:05.003535 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:05.003506    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:05.004406 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:05.004369    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-788n8\" (UniqueName: \"kubernetes.io/projected/d98cd19e-f8d9-4e07-b319-be194ea8f935-kube-api-access-788n8\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:05.095374 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:05.095337    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" event={"ID":"a2feec49-d795-4179-8e7b-cbb631a9b5e8","Type":"ContainerStarted","Data":"609e298c7f01f234faefe0ed000a1ec0a10ff6f34e274fd54c4abc5f63fb36d0"}
Oct 31 01:44:05.123772 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:05.123650    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podStartSLOduration=2.123624984 podStartE2EDuration="2.123624984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:05.119390864 +0000 UTC m=+1401.123105155" watchObservedRunningTime="2026-10-31 01:44:05.123624984 +0000 UTC m=+1401.127339276"
Oct 31 01:44:05.186865 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:05.186826    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:05.505493 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:44:05.504034    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98cd19e_f8d9_4e07_b319_be194ea8f935.slice/crio-1625fcf923c143a7269bfa8863463cb94fd6abdef87bc607bc950a42ddb2159a WatchSource:0}: Error finding container 1625fcf923c143a7269bfa8863463cb94fd6abdef87bc607bc950a42ddb2159a: Status 404 returned error can't find the container with id 1625fcf923c143a7269bfa8863463cb94fd6abdef87bc607bc950a42ddb2159a
Oct 31 01:44:05.508199 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:05.508162    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"]
Oct 31 01:44:05.513441 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:05.510261    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:44:05.513809 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:05.513775    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:44:05.513996 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:05.513977    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:44:05.680878 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:05.680821    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:05.958627 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:05.958589    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 01:44:05.958884 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:05.958854    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8" podUID="da0a52af-42a1-45d5-ab84-6696241d5489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 01:44:06.126154 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:06.126113    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d" event={"ID":"d98cd19e-f8d9-4e07-b319-be194ea8f935","Type":"ContainerStarted","Data":"402cb2a9e2e4de397a8dbbffffe55ca32740b06e8db69fa0998e1072a6bd0c3e"}
Oct 31 01:44:06.126577 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:06.126179    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d" event={"ID":"d98cd19e-f8d9-4e07-b319-be194ea8f935","Type":"ContainerStarted","Data":"1625fcf923c143a7269bfa8863463cb94fd6abdef87bc607bc950a42ddb2159a"}
Oct 31 01:44:06.154045 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:06.153950    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d" podStartSLOduration=2.153926514 podStartE2EDuration="2.153926514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:06.149600172 +0000 UTC m=+1402.153314465" watchObservedRunningTime="2026-10-31 01:44:06.153926514 +0000 UTC m=+1402.157640806"
Oct 31 01:44:06.187539 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:06.187488    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:06.188804 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:06.188773    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 31 01:44:06.189008 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:06.188973    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d" podUID="d98cd19e-f8d9-4e07-b319-be194ea8f935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 31 01:44:06.464491 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:06.464424    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:44:06.465314 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:06.465280    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:44:06.468726 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:06.468691    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:44:06.962740 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:06.962710    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:44:06.963002 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:06.962978    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:44:06.963705 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:06.963687    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:44:07.187604 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:07.187567    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 31 01:44:07.188114 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:07.187633    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d" podUID="d98cd19e-f8d9-4e07-b319-be194ea8f935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 31 01:44:08.203468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:08.201524    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 31 01:44:08.203468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:08.201585    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d" podUID="d98cd19e-f8d9-4e07-b319-be194ea8f935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 31 01:44:09.235473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.230805    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 31 01:44:09.235473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.230904    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d" podUID="d98cd19e-f8d9-4e07-b319-be194ea8f935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 31 01:44:09.555827 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.555744    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"]
Oct 31 01:44:09.559956 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.559937    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.573910 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.573879    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"]
Oct 31 01:44:09.587321 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.587281    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d872f42-06a3-4580-8930-fbe87eac3311-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.587521 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.587330    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d872f42-06a3-4580-8930-fbe87eac3311-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.587521 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.587415    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.587627 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.587550    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.587667 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.587631    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d872f42-06a3-4580-8930-fbe87eac3311-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.587754 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.587738    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4fxk\" (UniqueName: \"kubernetes.io/projected/2d872f42-06a3-4580-8930-fbe87eac3311-kube-api-access-p4fxk\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.587873 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.587857    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.587916 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.587888    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.587959 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.587917    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.631708 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.631673    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"]
Oct 31 01:44:09.637465 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.637420    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.638609 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.638584    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"]
Oct 31 01:44:09.689494 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.689435    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4fxk\" (UniqueName: \"kubernetes.io/projected/2d872f42-06a3-4580-8930-fbe87eac3311-kube-api-access-p4fxk\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.689705 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.689508    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg9p6\" (UniqueName: \"kubernetes.io/projected/214c7868-1e53-463f-8247-bed23603debd-kube-api-access-xg9p6\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.689705 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.689558    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.689705 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.689580    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.689705 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.689647    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.689929 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.689710    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/214c7868-1e53-463f-8247-bed23603debd-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.689929 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.689765    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.689929 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.689798    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/214c7868-1e53-463f-8247-bed23603debd-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.689929 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.689856    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/214c7868-1e53-463f-8247-bed23603debd-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.690125 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.689972    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.690125 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.690003    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.690125 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.690047    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.690125 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.690089    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.690267 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.690124    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d872f42-06a3-4580-8930-fbe87eac3311-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.690267 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.690144    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d872f42-06a3-4580-8930-fbe87eac3311-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.690267 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.690162    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.690267 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.690217    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.690267 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.690261    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.690478 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.690291    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d872f42-06a3-4580-8930-fbe87eac3311-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.690478 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.690334    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.690586 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.690564    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.690890 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.690868    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.690940 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.690916    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d872f42-06a3-4580-8930-fbe87eac3311-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.692111 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.692086    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.692576 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.692554    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d872f42-06a3-4580-8930-fbe87eac3311-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.728356 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.728289    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d872f42-06a3-4580-8930-fbe87eac3311-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.729173 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.729139    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4fxk\" (UniqueName: \"kubernetes.io/projected/2d872f42-06a3-4580-8930-fbe87eac3311-kube-api-access-p4fxk\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.791506 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.791469    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg9p6\" (UniqueName: \"kubernetes.io/projected/214c7868-1e53-463f-8247-bed23603debd-kube-api-access-xg9p6\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.791704 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.791563    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/214c7868-1e53-463f-8247-bed23603debd-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.791704 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.791595    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.791704 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.791614    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/214c7868-1e53-463f-8247-bed23603debd-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.791704 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.791632    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/214c7868-1e53-463f-8247-bed23603debd-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.791704 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.791671    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.791704 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.791699    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.792053 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.791792    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.792053 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.791828    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.792334 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.792307    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.792416 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.792349    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.792574 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.792547    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.792647 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.792587    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.792751 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.792730    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/214c7868-1e53-463f-8247-bed23603debd-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.794747 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.794727    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.794873 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.794850    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/214c7868-1e53-463f-8247-bed23603debd-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.837864 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.836892    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg9p6\" (UniqueName: \"kubernetes.io/projected/214c7868-1e53-463f-8247-bed23603debd-kube-api-access-xg9p6\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.839253 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.839132    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/214c7868-1e53-463f-8247-bed23603debd-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:09.877843 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.877813    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:09.956186 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:09.956144    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:10.187666 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:10.187584    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 31 01:44:10.187666 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:10.187638    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d" podUID="d98cd19e-f8d9-4e07-b319-be194ea8f935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 31 01:44:10.248966 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:10.248925    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:44:10.250192 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:10.250165    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:44:10.250362 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:10.250345    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:44:10.258116 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:10.254712    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"]
Oct 31 01:44:10.283373 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:10.283343    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"]
Oct 31 01:44:10.291608 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:44:10.291570    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214c7868_1e53_463f_8247_bed23603debd.slice/crio-662f85f1b47d9491fbd41c3f49e8c6d7ccc741ca916e14ccce1f4715acc474ba WatchSource:0}: Error finding container 662f85f1b47d9491fbd41c3f49e8c6d7ccc741ca916e14ccce1f4715acc474ba: Status 404 returned error can't find the container with id 662f85f1b47d9491fbd41c3f49e8c6d7ccc741ca916e14ccce1f4715acc474ba
Oct 31 01:44:10.293612 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:10.293570    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:44:10.293828 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:10.293802    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:44:10.293963 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:10.293942    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:44:10.297776 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:10.297754    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl" event={"ID":"2d872f42-06a3-4580-8930-fbe87eac3311","Type":"ContainerStarted","Data":"8526a12bfc80aaded94216dd8a491039a2c64390a936f5290876ec23fc3155cf"}
Oct 31 01:44:11.188208 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:11.188164    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 31 01:44:11.188432 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:11.188224    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d" podUID="d98cd19e-f8d9-4e07-b319-be194ea8f935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 31 01:44:11.358343 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:11.358292    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2" event={"ID":"214c7868-1e53-463f-8247-bed23603debd","Type":"ContainerStarted","Data":"a73b3a3e25c8fc6a3df26a76791f6ccf7ad5e2ed362aaefb3417df8b4f789cb2"}
Oct 31 01:44:11.359008 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:11.358978    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2" event={"ID":"214c7868-1e53-463f-8247-bed23603debd","Type":"ContainerStarted","Data":"662f85f1b47d9491fbd41c3f49e8c6d7ccc741ca916e14ccce1f4715acc474ba"}
Oct 31 01:44:11.382954 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:11.382916    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl" event={"ID":"2d872f42-06a3-4580-8930-fbe87eac3311","Type":"ContainerStarted","Data":"68180ed21c305f7c9115e6be1961d5964b92606e157d41f5824f60ac86c14902"}
Oct 31 01:44:11.458469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:11.457495    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2" podStartSLOduration=2.457472544 podStartE2EDuration="2.457472544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:11.408080854 +0000 UTC m=+1407.411795148" watchObservedRunningTime="2026-10-31 01:44:11.457472544 +0000 UTC m=+1407.461186826"
Oct 31 01:44:11.480498 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:11.469490    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl" podStartSLOduration=2.469465644 podStartE2EDuration="2.469465644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:11.453941907 +0000 UTC m=+1407.457656201" watchObservedRunningTime="2026-10-31 01:44:11.469465644 +0000 UTC m=+1407.473179936"
Oct 31 01:44:11.879758 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:11.878866    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:11.880998 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:11.880963    5650 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 01:44:11.881487 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:11.881179    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl" podUID="2d872f42-06a3-4580-8930-fbe87eac3311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 01:44:11.960469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:11.956839    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:11.960655 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:11.960583    5650 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:44:11.960742 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:11.960647    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2" podUID="214c7868-1e53-463f-8247-bed23603debd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 01:44:12.188054 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:12.187968    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 31 01:44:12.188054 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:12.188033    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d" podUID="d98cd19e-f8d9-4e07-b319-be194ea8f935" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 31 01:44:12.878264 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:12.878232    5650 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 01:44:12.878662 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:12.878284    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl" podUID="2d872f42-06a3-4580-8930-fbe87eac3311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 01:44:12.956897 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:12.956858    5650 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:44:12.957228 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:12.957195    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2" podUID="214c7868-1e53-463f-8247-bed23603debd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 01:44:13.683311 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:13.680905    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:13.748473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:13.735977    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:13.778175 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:13.778139    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:13.779207 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:13.779081    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:13.780686 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:13.780353    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 01:44:13.878314 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:13.878272    5650 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 01:44:13.878796 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:13.878331    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl" podUID="2d872f42-06a3-4580-8930-fbe87eac3311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 01:44:13.958493 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:13.958437    5650 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:44:13.958679 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:13.958520    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2" podUID="214c7868-1e53-463f-8247-bed23603debd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 01:44:14.466288 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:14.466109    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:14.878816 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:14.878719    5650 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 01:44:14.878816 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:14.878784    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl" podUID="2d872f42-06a3-4580-8930-fbe87eac3311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 01:44:14.956764 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:14.956723    5650 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:44:14.956940 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:14.956776    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2" podUID="214c7868-1e53-463f-8247-bed23603debd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 01:44:15.879257 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:15.879219    5650 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 01:44:15.879745 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:15.879293    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl" podUID="2d872f42-06a3-4580-8930-fbe87eac3311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 01:44:15.956891 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:15.956843    5650 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:44:15.957063 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:15.956903    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2" podUID="214c7868-1e53-463f-8247-bed23603debd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 01:44:16.882473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:16.878935    5650 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 01:44:16.882473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:16.879000    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl" podUID="2d872f42-06a3-4580-8930-fbe87eac3311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 01:44:16.956841 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:16.956798    5650 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:44:16.956994 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:16.956885    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2" podUID="214c7868-1e53-463f-8247-bed23603debd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 01:44:18.225469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:18.217418    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:18.225469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:18.218028    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:18.225469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:18.223806    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 01:44:18.250468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:18.249524    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:18.250468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:18.250204    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:18.252125 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:18.252098    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 01:44:20.832469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:20.831964    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:21.031625 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:21.028546    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:21.117703 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:21.117595    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:21.341584 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:21.341532    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:21.421544 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:21.420549    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:21.459474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:21.454871    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 01:44:21.459474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:21.456078    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:21.528639 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:21.528584    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:21.561473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:21.535527    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e3532af-4nsks"]
Oct 31 01:44:21.563553 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:21.563520    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e3532af-4nsks"
Oct 31 01:44:21.569750 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:21.569713    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e3532af-4nsks"]
Oct 31 01:44:21.578727 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:21.578697    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:44:21.579075 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:21.579039    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:44:21.753999 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:21.753786    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a681024-af72-4506-a3ca-97e20cfe4e1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e3532af-4nsks\" (UID: \"3a681024-af72-4506-a3ca-97e20cfe4e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e3532af-4nsks"
Oct 31 01:44:21.973471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:21.944303    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 01:44:22.025613 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.015631    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 31 01:44:22.025613 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.016318    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:44:22.046477 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.035577    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a681024-af72-4506-a3ca-97e20cfe4e1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e3532af-4nsks\" (UID: \"3a681024-af72-4506-a3ca-97e20cfe4e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e3532af-4nsks"
Oct 31 01:44:22.046477 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.035663    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bp8j\" (UniqueName: \"kubernetes.io/projected/3a681024-af72-4506-a3ca-97e20cfe4e1c-kube-api-access-7bp8j\") pod \"maas-api-revoke-sub-9e3532af-4nsks\" (UID: \"3a681024-af72-4506-a3ca-97e20cfe4e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e3532af-4nsks"
Oct 31 01:44:22.046477 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.036419    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 01:44:22.132485 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.128789    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 01:44:22.132485 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.128951    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:22.132485 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.129061    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:44:22.145654 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.143660    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a681024-af72-4506-a3ca-97e20cfe4e1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e3532af-4nsks\" (UID: \"3a681024-af72-4506-a3ca-97e20cfe4e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e3532af-4nsks"
Oct 31 01:44:22.145654 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.143741    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bp8j\" (UniqueName: \"kubernetes.io/projected/3a681024-af72-4506-a3ca-97e20cfe4e1c-kube-api-access-7bp8j\") pod \"maas-api-revoke-sub-9e3532af-4nsks\" (UID: \"3a681024-af72-4506-a3ca-97e20cfe4e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e3532af-4nsks"
Oct 31 01:44:22.145654 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.143910    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a681024-af72-4506-a3ca-97e20cfe4e1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e3532af-4nsks\" (UID: \"3a681024-af72-4506-a3ca-97e20cfe4e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e3532af-4nsks"
Oct 31 01:44:22.145654 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.144992    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a681024-af72-4506-a3ca-97e20cfe4e1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e3532af-4nsks\" (UID: \"3a681024-af72-4506-a3ca-97e20cfe4e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e3532af-4nsks"
Oct 31 01:44:22.170614 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.170569    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:44:22.170885 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.170851    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7" podUID="dedb244a-f01c-463e-8201-c6d0ad8e3f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:22.373395 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.347531    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:22.436639 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.436591    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Oct 31 01:44:22.455476 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.452807    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a681024-af72-4506-a3ca-97e20cfe4e1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e3532af-4nsks\" (UID: \"3a681024-af72-4506-a3ca-97e20cfe4e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e3532af-4nsks"
Oct 31 01:44:22.554669 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.554612    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:22.556790 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.556661    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:22.789717 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.727160    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 01:44:22.790140 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.790116    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bp8j\" (UniqueName: \"kubernetes.io/projected/3a681024-af72-4506-a3ca-97e20cfe4e1c-kube-api-access-7bp8j\") pod \"maas-api-revoke-sub-9e3532af-4nsks\" (UID: \"3a681024-af72-4506-a3ca-97e20cfe4e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e3532af-4nsks"
Oct 31 01:44:22.813677 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.813645    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e3532af-4nsks"
Oct 31 01:44:22.840733 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.840697    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:44:22.840991 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.840973    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:22.841178 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.841161    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:44:22.841335 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.841321    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:44:22.841483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:22.841468    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:44:23.182402 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:23.182336    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 31 01:44:23.217523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:23.193520    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 01:44:23.217523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:23.214745    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:44:23.237885 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:23.237835    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:23.264467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:23.260737    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 01:44:23.264467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:23.262109    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:44:23.327150 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:23.327122    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:44:23.349536 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:23.349506    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:44:23.403272 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:23.403078    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:44:23.404071 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:23.404048    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:44:23.419587 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:23.419546    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:23.423853 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:23.423638    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded"
Oct 31 01:44:23.446157 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:23.446121    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:44:23.447231 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:23.447199    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:23.497405 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:23.469437    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:44:23.515575 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:23.515527    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:44:23.518221 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:23.518198    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:23.555491 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:23.555461    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:44:23.766771 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:23.766740    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e3532af-4nsks"]
Oct 31 01:44:23.766928 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:44:23.766858    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a681024_af72_4506_a3ca_97e20cfe4e1c.slice/crio-b5d5ce40fbdae2308883c0e44a1c534331f494e7400c576ac02539ac06d457be WatchSource:0}: Error finding container b5d5ce40fbdae2308883c0e44a1c534331f494e7400c576ac02539ac06d457be: Status 404 returned error can't find the container with id b5d5ce40fbdae2308883c0e44a1c534331f494e7400c576ac02539ac06d457be
Oct 31 01:44:24.437796 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:24.437693    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e3532af-4nsks" event={"ID":"3a681024-af72-4506-a3ca-97e20cfe4e1c","Type":"ContainerStarted","Data":"f60e22db9a1494bb66b85c62017c24dfb5c5b05accb9b440ed326d534ff6efd7"}
Oct 31 01:44:24.437796 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:24.437761    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e3532af-4nsks" event={"ID":"3a681024-af72-4506-a3ca-97e20cfe4e1c","Type":"ContainerStarted","Data":"b5d5ce40fbdae2308883c0e44a1c534331f494e7400c576ac02539ac06d457be"}
Oct 31 01:44:24.461093 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:24.461036    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e3532af-4nsks" podStartSLOduration=3.461022833 podStartE2EDuration="3.461022833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:24.458050163 +0000 UTC m=+1420.461764455" watchObservedRunningTime="2026-10-31 01:44:24.461022833 +0000 UTC m=+1420.464737123"
Oct 31 01:44:25.445836 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:25.445797    5650 generic.go:356] "Generic (PLEG): container finished" podID="3a681024-af72-4506-a3ca-97e20cfe4e1c" containerID="f60e22db9a1494bb66b85c62017c24dfb5c5b05accb9b440ed326d534ff6efd7" exitCode=0
Oct 31 01:44:25.446262 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:25.445891    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e3532af-4nsks" event={"ID":"3a681024-af72-4506-a3ca-97e20cfe4e1c","Type":"ContainerDied","Data":"f60e22db9a1494bb66b85c62017c24dfb5c5b05accb9b440ed326d534ff6efd7"}
Oct 31 01:44:26.611341 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:26.611312    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e3532af-4nsks"
Oct 31 01:44:26.679925 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:26.679892    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a681024-af72-4506-a3ca-97e20cfe4e1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a681024-af72-4506-a3ca-97e20cfe4e1c-maas-api-xks-ca\") pod \"3a681024-af72-4506-a3ca-97e20cfe4e1c\" (UID: \"3a681024-af72-4506-a3ca-97e20cfe4e1c\") "
Oct 31 01:44:26.680104 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:26.679960    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a681024-af72-4506-a3ca-97e20cfe4e1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a681024-af72-4506-a3ca-97e20cfe4e1c-maas-api-openshift-ca\") pod \"3a681024-af72-4506-a3ca-97e20cfe4e1c\" (UID: \"3a681024-af72-4506-a3ca-97e20cfe4e1c\") "
Oct 31 01:44:26.680104 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:26.680063    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a681024-af72-4506-a3ca-97e20cfe4e1c-kube-api-access-7bp8j\" (UniqueName: \"kubernetes.io/projected/3a681024-af72-4506-a3ca-97e20cfe4e1c-kube-api-access-7bp8j\") pod \"3a681024-af72-4506-a3ca-97e20cfe4e1c\" (UID: \"3a681024-af72-4506-a3ca-97e20cfe4e1c\") "
Oct 31 01:44:26.680466 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:26.680421    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a681024-af72-4506-a3ca-97e20cfe4e1c-maas-api-openshift-ca" pod "3a681024-af72-4506-a3ca-97e20cfe4e1c" (UID: "3a681024-af72-4506-a3ca-97e20cfe4e1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:26.680738 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:26.680714    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a681024-af72-4506-a3ca-97e20cfe4e1c-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:26.682752 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:26.682722    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a681024-af72-4506-a3ca-97e20cfe4e1c-maas-api-xks-ca" pod "3a681024-af72-4506-a3ca-97e20cfe4e1c" (UID: "3a681024-af72-4506-a3ca-97e20cfe4e1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:44:26.682842 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:26.682727    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a681024-af72-4506-a3ca-97e20cfe4e1c-kube-api-access-7bp8j" pod "3a681024-af72-4506-a3ca-97e20cfe4e1c" (UID: "3a681024-af72-4506-a3ca-97e20cfe4e1c"). InnerVolumeSpecName "kube-api-access-7bp8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:26.782385 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:26.782281    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bp8j\" (UniqueName: \"kubernetes.io/projected/3a681024-af72-4506-a3ca-97e20cfe4e1c-kube-api-access-7bp8j\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:26.782385 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:26.782333    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a681024-af72-4506-a3ca-97e20cfe4e1c-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:27.462408 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:27.462381    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e3532af-4nsks"
Oct 31 01:44:27.462645 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:27.462371    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e3532af-4nsks" event={"ID":"3a681024-af72-4506-a3ca-97e20cfe4e1c","Type":"ContainerDied","Data":"b5d5ce40fbdae2308883c0e44a1c534331f494e7400c576ac02539ac06d457be"}
Oct 31 01:44:27.462645 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:27.462519    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5d5ce40fbdae2308883c0e44a1c534331f494e7400c576ac02539ac06d457be"
Oct 31 01:44:28.553944 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:44:28.553905    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:44:34.062473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:34.059610    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:34.331767 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:34.331626    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:34.354644 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:34.354548    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:34.413468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:34.412632    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 01:44:34.414179 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:34.414137    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:34.430630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:34.430577    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:34.480473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:34.479728    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:34.523496 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:34.521537    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 31 01:44:34.540486 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:34.536768    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Oct 31 01:44:34.596547 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:34.592757    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:34.741095 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:34.741046    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:34.969687 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:34.969636    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:35.375832 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:35.335621    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 01:44:35.375832 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:35.336264    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:44:35.375832 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:35.357577    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:35.375832 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:35.358026    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:44:35.440481 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:35.416310    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:35.441252 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:35.435830    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:35.449953 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:35.435910    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:35.556695 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:35.441930    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:44:35.557064 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:35.557040    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:44:35.557307 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:35.557277    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:44:35.571566 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:35.554828    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 31 01:44:35.573297 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:35.573256    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:44:35.596273 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:35.496959    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:35.651985 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:35.554766    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:35.677012 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:35.661270    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:44:35.677012 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:35.661346    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:35.677012 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:35.623042    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 01:44:35.716169 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:35.714820    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:44:35.953554 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:35.948557    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:44:35.953554 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:35.948629    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4" podUID="d2118bab-9cba-42f7-814d-730046ef72a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:36.405529 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:36.377110    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:36.405529 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:36.387189    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 01:44:36.468890 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:36.468831    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:36.554518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:36.494584    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-28qvk" podUID="1548a9c1-776d-41f4-addb-399ab46f60c0" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.24:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:36.567014 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:36.541421    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:36.602947 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:36.602902    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 01:44:36.680656 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:36.669626    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:36.820650 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:36.672161    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:36.821330 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:36.713696    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 01:44:36.821614 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:36.753718    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded"
Oct 31 01:44:36.821708 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:36.688699    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:37.151472 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:44:37.145647    5650 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.035s"
Oct 31 01:44:37.338472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:37.329522    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:44:37.338472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:37.329617    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7" podUID="dedb244a-f01c-463e-8201-c6d0ad8e3f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:37.379938 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:37.379884    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:37.414055 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:37.413923    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 01:44:37.493471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:37.486101    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:37.578496 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:37.570571    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 01:44:37.616185 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:37.616127    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:37.665061 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:37.658859    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:44:37.665061 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:37.658930    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx" podUID="ebc90b15-cef7-4483-a8bb-1a33950f9fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:37.820594 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:37.820541    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:37.840472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:37.820844    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 31 01:44:37.842275 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:37.842235    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:37.843213 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:37.842782    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:38.008061 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:38.008017    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:44:38.008280 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:38.008087    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8" podUID="da0a52af-42a1-45d5-ab84-6696241d5489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:38.167006 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:38.166560    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded"
Oct 31 01:44:38.390297 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:38.389547    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:38.417196 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:38.415968    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:38.488065 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:38.486763    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:44:38.587469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:38.583559    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:38.621472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:38.617832    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:38.780565 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:38.779410    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:38.793483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:38.793434    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:44:38.802436 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:38.802205    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:44:38.808847 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:38.806711    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:44:38.817931 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:38.817805    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:44:38.821415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:38.821377    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:38.841364 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:38.834488    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:44:38.880198 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:38.880167    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:44:38.887676 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:38.887642    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:38.888265 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:38.888242    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:44:41.010558 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:41.010509    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49e8ad58-2dspg"]
Oct 31 01:44:41.011106 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:41.011091    5650 cpu_manager.go:414] "containerMap: removing container" podUID="3a681024-af72-4506-a3ca-97e20cfe4e1c" containerName="revoke-keys"
Oct 31 01:44:41.011150 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:41.011107    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a681024-af72-4506-a3ca-97e20cfe4e1c" containerName="revoke-keys"
Oct 31 01:44:41.011224 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:41.011216    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a681024-af72-4506-a3ca-97e20cfe4e1c" containerName="revoke-keys"
Oct 31 01:44:41.014941 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:41.014920    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e8ad58-2dspg"
Oct 31 01:44:41.023288 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:41.023261    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49e8ad58-2dspg"]
Oct 31 01:44:41.023665 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:41.023634    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:44:41.023904 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:41.023884    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:44:41.115639 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:41.115599    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4vrg\" (UniqueName: \"kubernetes.io/projected/3ecc2be1-1e13-4564-b18d-8256a69cf828-kube-api-access-h4vrg\") pod \"maas-api-revoke-sub-49e8ad58-2dspg\" (UID: \"3ecc2be1-1e13-4564-b18d-8256a69cf828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e8ad58-2dspg"
Oct 31 01:44:41.115844 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:41.115684    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ecc2be1-1e13-4564-b18d-8256a69cf828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49e8ad58-2dspg\" (UID: \"3ecc2be1-1e13-4564-b18d-8256a69cf828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e8ad58-2dspg"
Oct 31 01:44:41.115844 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:41.115792    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ecc2be1-1e13-4564-b18d-8256a69cf828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49e8ad58-2dspg\" (UID: \"3ecc2be1-1e13-4564-b18d-8256a69cf828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e8ad58-2dspg"
Oct 31 01:44:41.217117 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:41.217085    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4vrg\" (UniqueName: \"kubernetes.io/projected/3ecc2be1-1e13-4564-b18d-8256a69cf828-kube-api-access-h4vrg\") pod \"maas-api-revoke-sub-49e8ad58-2dspg\" (UID: \"3ecc2be1-1e13-4564-b18d-8256a69cf828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e8ad58-2dspg"
Oct 31 01:44:41.217291 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:41.217166    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ecc2be1-1e13-4564-b18d-8256a69cf828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49e8ad58-2dspg\" (UID: \"3ecc2be1-1e13-4564-b18d-8256a69cf828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e8ad58-2dspg"
Oct 31 01:44:41.217291 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:41.217250    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ecc2be1-1e13-4564-b18d-8256a69cf828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49e8ad58-2dspg\" (UID: \"3ecc2be1-1e13-4564-b18d-8256a69cf828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e8ad58-2dspg"
Oct 31 01:44:41.218012 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:41.217975    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ecc2be1-1e13-4564-b18d-8256a69cf828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49e8ad58-2dspg\" (UID: \"3ecc2be1-1e13-4564-b18d-8256a69cf828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e8ad58-2dspg"
Oct 31 01:44:41.219822 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:41.219797    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ecc2be1-1e13-4564-b18d-8256a69cf828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49e8ad58-2dspg\" (UID: \"3ecc2be1-1e13-4564-b18d-8256a69cf828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e8ad58-2dspg"
Oct 31 01:44:41.230407 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:41.230377    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4vrg\" (UniqueName: \"kubernetes.io/projected/3ecc2be1-1e13-4564-b18d-8256a69cf828-kube-api-access-h4vrg\") pod \"maas-api-revoke-sub-49e8ad58-2dspg\" (UID: \"3ecc2be1-1e13-4564-b18d-8256a69cf828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e8ad58-2dspg"
Oct 31 01:44:41.335316 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:41.335197    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e8ad58-2dspg"
Oct 31 01:44:41.636395 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:41.636308    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49e8ad58-2dspg"]
Oct 31 01:44:42.436612 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:42.436582    5650 generic.go:356] "Generic (PLEG): container finished" podID="3ecc2be1-1e13-4564-b18d-8256a69cf828" containerID="37d967cd3f2e322e95aeb7e17daf9bc05dac826cce952fef71a4b9fdc748662e" exitCode=0
Oct 31 01:44:42.436993 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:42.436667    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e8ad58-2dspg" event={"ID":"3ecc2be1-1e13-4564-b18d-8256a69cf828","Type":"ContainerDied","Data":"37d967cd3f2e322e95aeb7e17daf9bc05dac826cce952fef71a4b9fdc748662e"}
Oct 31 01:44:42.436993 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:42.436723    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e8ad58-2dspg" event={"ID":"3ecc2be1-1e13-4564-b18d-8256a69cf828","Type":"ContainerStarted","Data":"a1cbadceda9e85b3f7152456d0bec34e851b6405fa990d3b404f589f5ed38079"}
Oct 31 01:44:43.609048 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:43.609020    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e8ad58-2dspg"
Oct 31 01:44:43.652247 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:43.652217    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ecc2be1-1e13-4564-b18d-8256a69cf828-kube-api-access-h4vrg\" (UniqueName: \"kubernetes.io/projected/3ecc2be1-1e13-4564-b18d-8256a69cf828-kube-api-access-h4vrg\") pod \"3ecc2be1-1e13-4564-b18d-8256a69cf828\" (UID: \"3ecc2be1-1e13-4564-b18d-8256a69cf828\") "
Oct 31 01:44:43.652517 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:43.652493    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ecc2be1-1e13-4564-b18d-8256a69cf828-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ecc2be1-1e13-4564-b18d-8256a69cf828-maas-api-openshift-ca\") pod \"3ecc2be1-1e13-4564-b18d-8256a69cf828\" (UID: \"3ecc2be1-1e13-4564-b18d-8256a69cf828\") "
Oct 31 01:44:43.652614 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:43.652587    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ecc2be1-1e13-4564-b18d-8256a69cf828-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ecc2be1-1e13-4564-b18d-8256a69cf828-maas-api-xks-ca\") pod \"3ecc2be1-1e13-4564-b18d-8256a69cf828\" (UID: \"3ecc2be1-1e13-4564-b18d-8256a69cf828\") "
Oct 31 01:44:43.652856 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:43.652829    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ecc2be1-1e13-4564-b18d-8256a69cf828-maas-api-openshift-ca" pod "3ecc2be1-1e13-4564-b18d-8256a69cf828" (UID: "3ecc2be1-1e13-4564-b18d-8256a69cf828"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:43.653236 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:43.653211    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ecc2be1-1e13-4564-b18d-8256a69cf828-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:43.655021 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:43.654996    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ecc2be1-1e13-4564-b18d-8256a69cf828-maas-api-xks-ca" pod "3ecc2be1-1e13-4564-b18d-8256a69cf828" (UID: "3ecc2be1-1e13-4564-b18d-8256a69cf828"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:44:43.655149 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:43.655015    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ecc2be1-1e13-4564-b18d-8256a69cf828-kube-api-access-h4vrg" pod "3ecc2be1-1e13-4564-b18d-8256a69cf828" (UID: "3ecc2be1-1e13-4564-b18d-8256a69cf828"). InnerVolumeSpecName "kube-api-access-h4vrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:43.754687 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:43.754647    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4vrg\" (UniqueName: \"kubernetes.io/projected/3ecc2be1-1e13-4564-b18d-8256a69cf828-kube-api-access-h4vrg\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:43.754687 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:43.754691    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ecc2be1-1e13-4564-b18d-8256a69cf828-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:44:44.451652 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:44.451571    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e8ad58-2dspg"
Oct 31 01:44:44.451652 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:44.451611    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49e8ad58-2dspg" event={"ID":"3ecc2be1-1e13-4564-b18d-8256a69cf828","Type":"ContainerDied","Data":"a1cbadceda9e85b3f7152456d0bec34e851b6405fa990d3b404f589f5ed38079"}
Oct 31 01:44:44.451840 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:44.451661    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1cbadceda9e85b3f7152456d0bec34e851b6405fa990d3b404f589f5ed38079"
Oct 31 01:44:49.984900 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:49.984865    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"]
Oct 31 01:44:49.985485 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:49.985467    5650 cpu_manager.go:414] "containerMap: removing container" podUID="3ecc2be1-1e13-4564-b18d-8256a69cf828" containerName="revoke-keys"
Oct 31 01:44:49.985485 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:49.985483    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ecc2be1-1e13-4564-b18d-8256a69cf828" containerName="revoke-keys"
Oct 31 01:44:49.985619 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:49.985608    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ecc2be1-1e13-4564-b18d-8256a69cf828" containerName="revoke-keys"
Oct 31 01:44:49.990716 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:49.990690    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.001424 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.001398    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"]
Oct 31 01:44:50.048704 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.048664    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.048887 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.048740    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.048887 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.048835    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdfkk\" (UniqueName: \"kubernetes.io/projected/07620cf8-e211-41cf-8faf-177eeb1e2cac-kube-api-access-xdfkk\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.048887 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.048866    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.049051 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.049019    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.049142 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.049121    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.049184 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.049160    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07620cf8-e211-41cf-8faf-177eeb1e2cac-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.049261 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.049243    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.049353 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.049340    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.150748 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.150711    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.150748 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.150751    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07620cf8-e211-41cf-8faf-177eeb1e2cac-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.150988 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.150776    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.150988 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.150826    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.150988 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.150883    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.150988 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.150936    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.151213 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.150997    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdfkk\" (UniqueName: \"kubernetes.io/projected/07620cf8-e211-41cf-8faf-177eeb1e2cac-kube-api-access-xdfkk\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.151213 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.151033    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.151213 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.151121    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.151213 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.151166    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.151416 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.151246    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.151507 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.151485    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.151548 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.151494    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.152200 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.152174    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07620cf8-e211-41cf-8faf-177eeb1e2cac-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.153370 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.153342    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.153727 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.153705    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.170588 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.170558    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.170719 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.170557    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdfkk\" (UniqueName: \"kubernetes.io/projected/07620cf8-e211-41cf-8faf-177eeb1e2cac-kube-api-access-xdfkk\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.309609 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.309507    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:50.606606 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.606574    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"]
Oct 31 01:44:50.610293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.610255    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:44:50.610416 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.610336    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:44:50.610416 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:50.610364    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:44:51.580472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:51.578233    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5" event={"ID":"07620cf8-e211-41cf-8faf-177eeb1e2cac","Type":"ContainerStarted","Data":"acc8ac22b724b78683178db63b2fdb0e24da950588fbaade22e7cbd24cbf9846"}
Oct 31 01:44:51.580472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:51.578304    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5" event={"ID":"07620cf8-e211-41cf-8faf-177eeb1e2cac","Type":"ContainerStarted","Data":"af51d592371d9b8d7ad52d6d2e23363caa00b61a02f6635ffcff6a19ca0cd33d"}
Oct 31 01:44:51.655472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:51.650830    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5" podStartSLOduration=2.65080735 podStartE2EDuration="2.65080735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:51.631952699 +0000 UTC m=+1447.635666992" watchObservedRunningTime="2026-10-31 01:44:51.65080735 +0000 UTC m=+1447.654521642"
Oct 31 01:44:52.314205 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:52.314165    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:44:52.318663 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:52.318624    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 01:44:52.318958 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:52.318926    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5" podUID="07620cf8-e211-41cf-8faf-177eeb1e2cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 01:44:52.500471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:52.498545    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:52.577532 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:52.576402    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:52.657470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:52.656562    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:52.792471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:52.791769    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:52.825471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:52.824561    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 01:44:52.825471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:52.825092    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:52.872532 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:52.871687    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:52.884472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:52.881678    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:52.903532 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:52.895297    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:53.323484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:53.320877    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 01:44:53.323484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:53.320943    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5" podUID="07620cf8-e211-41cf-8faf-177eeb1e2cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 01:44:53.398482 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:53.396541    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded"
Oct 31 01:44:53.568478 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:53.558933    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:53.749585 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:53.569102    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:44:53.787567 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:53.597287    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:53.787567 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:53.673618    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 01:44:53.787567 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:53.785570    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:44:53.787567 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:53.785628    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:44:53.872399 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:53.872348    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:53.882496 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:53.874785    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:44:53.906473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:53.906332    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 01:44:53.910470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:53.908712    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 01:44:53.910470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:53.910038    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 01:44:53.910470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:53.910261    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:54.128109 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:53.918524    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:44:54.128109 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:53.918652    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:44:54.128109 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:53.918703    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:44:54.128109 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:53.918742    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:54.128109 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:53.935560    5650 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:44:54.128109 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:53.935627    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8" podUID="af0ee471-3ebf-426f-88e8-689ddc123d8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:54.128109 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:53.937644    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 31 01:44:54.128109 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:53.938025    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:44:54.195494 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:54.130240    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded"
Oct 31 01:44:54.377868 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:54.369232    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 01:44:54.377868 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:54.369312    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5" podUID="07620cf8-e211-41cf-8faf-177eeb1e2cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 01:44:54.579100 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:54.579054    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 01:44:54.777506 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:54.695628    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:54.798695 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:54.786572    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 01:44:54.812328 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:54.799164    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:54.915659 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:54.915543    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 01:44:54.932357 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:54.932310    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 01:44:54.964175 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:54.953039    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 31 01:44:54.964175 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:54.954363    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 01:44:54.964175 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:54.955009    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 01:44:54.964175 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:54.956012    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Oct 31 01:44:55.012474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:54.976763    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:55.330063 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:55.330026    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 01:44:55.355575 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:55.355523    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5" podUID="07620cf8-e211-41cf-8faf-177eeb1e2cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 01:44:55.789493 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:55.784884    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:55.844469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:55.827280    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:55.856472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:55.834241    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:56.013383 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:55.929746    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:56.026795 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:55.958096    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Oct 31 01:44:56.065532 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:55.958151    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 31 01:44:56.067291 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:55.958173    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 01:44:56.090169 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:55.958219    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 01:44:56.202512 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:55.958289    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 01:44:56.336039 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.335932    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-28qvk" podUID="1548a9c1-776d-41f4-addb-399ab46f60c0" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.24:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:44:56.368025 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.367962    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 01:44:56.368365 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.368327    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5" podUID="07620cf8-e211-41cf-8faf-177eeb1e2cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 01:44:56.491838 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.491806    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"]
Oct 31 01:44:56.503670 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.503640    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"]
Oct 31 01:44:56.503905 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.503892    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64bf97a5-t6vfn"]
Oct 31 01:44:56.504467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.503786    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.523477 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.513655    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64bf97a5-t6vfn"]
Oct 31 01:44:56.523477 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.513785    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64bf97a5-t6vfn"
Oct 31 01:44:56.523477 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.521075    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:44:56.523477 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.521872    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:44:56.584435 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.583196    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:44:56.613760 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.613605    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:44:56.614878 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.614852    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:44:56.615106 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.615093    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:44:56.618293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.618268    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:44:56.621805 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.621784    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:44:56.631518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.626825    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/683dbaa0-6bdd-4431-8b3b-6ca550a9f509-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64bf97a5-t6vfn\" (UID: \"683dbaa0-6bdd-4431-8b3b-6ca550a9f509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64bf97a5-t6vfn"
Oct 31 01:44:56.631518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.626864    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/683dbaa0-6bdd-4431-8b3b-6ca550a9f509-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64bf97a5-t6vfn\" (UID: \"683dbaa0-6bdd-4431-8b3b-6ca550a9f509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64bf97a5-t6vfn"
Oct 31 01:44:56.631518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.626904    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.631518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.627026    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.631518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.627079    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da800208-fbd5-460f-b1cc-a212a4445291-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.631518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.627133    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.631518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.627170    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da800208-fbd5-460f-b1cc-a212a4445291-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.631518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.627202    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqmrv\" (UniqueName: \"kubernetes.io/projected/683dbaa0-6bdd-4431-8b3b-6ca550a9f509-kube-api-access-tqmrv\") pod \"maas-api-revoke-sub-64bf97a5-t6vfn\" (UID: \"683dbaa0-6bdd-4431-8b3b-6ca550a9f509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64bf97a5-t6vfn"
Oct 31 01:44:56.631518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.627254    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da800208-fbd5-460f-b1cc-a212a4445291-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.631518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.627283    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.631518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.627341    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnpvm\" (UniqueName: \"kubernetes.io/projected/da800208-fbd5-460f-b1cc-a212a4445291-kube-api-access-rnpvm\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.631518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.627369    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.647073 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.644035    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:44:56.647073 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.644324    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:44:56.663867 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.662851    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:44:56.744982 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.744943    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.745161 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.745055    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da800208-fbd5-460f-b1cc-a212a4445291-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.745218 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.745198    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.745288 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.745273    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da800208-fbd5-460f-b1cc-a212a4445291-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.745357 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.745342    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqmrv\" (UniqueName: \"kubernetes.io/projected/683dbaa0-6bdd-4431-8b3b-6ca550a9f509-kube-api-access-tqmrv\") pod \"maas-api-revoke-sub-64bf97a5-t6vfn\" (UID: \"683dbaa0-6bdd-4431-8b3b-6ca550a9f509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64bf97a5-t6vfn"
Oct 31 01:44:56.745491 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.745472    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da800208-fbd5-460f-b1cc-a212a4445291-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.745567 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.745512    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.745634 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.745617    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnpvm\" (UniqueName: \"kubernetes.io/projected/da800208-fbd5-460f-b1cc-a212a4445291-kube-api-access-rnpvm\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.745700 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.745656    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.745879 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.745854    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/683dbaa0-6bdd-4431-8b3b-6ca550a9f509-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64bf97a5-t6vfn\" (UID: \"683dbaa0-6bdd-4431-8b3b-6ca550a9f509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64bf97a5-t6vfn"
Oct 31 01:44:56.745944 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.745895    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/683dbaa0-6bdd-4431-8b3b-6ca550a9f509-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64bf97a5-t6vfn\" (UID: \"683dbaa0-6bdd-4431-8b3b-6ca550a9f509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64bf97a5-t6vfn"
Oct 31 01:44:56.746020 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.745989    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.746081 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.746028    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da800208-fbd5-460f-b1cc-a212a4445291-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.746655 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.746628    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.746755 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.746661    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.746968 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.746943    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.747933 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.747206    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.748060 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.747946    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/683dbaa0-6bdd-4431-8b3b-6ca550a9f509-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64bf97a5-t6vfn\" (UID: \"683dbaa0-6bdd-4431-8b3b-6ca550a9f509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64bf97a5-t6vfn"
Oct 31 01:44:56.750289 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.750260    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.750370 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.750316    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da800208-fbd5-460f-b1cc-a212a4445291-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.751573 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.751557    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/683dbaa0-6bdd-4431-8b3b-6ca550a9f509-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64bf97a5-t6vfn\" (UID: \"683dbaa0-6bdd-4431-8b3b-6ca550a9f509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64bf97a5-t6vfn"
Oct 31 01:44:56.762548 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.762503    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqmrv\" (UniqueName: \"kubernetes.io/projected/683dbaa0-6bdd-4431-8b3b-6ca550a9f509-kube-api-access-tqmrv\") pod \"maas-api-revoke-sub-64bf97a5-t6vfn\" (UID: \"683dbaa0-6bdd-4431-8b3b-6ca550a9f509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64bf97a5-t6vfn"
Oct 31 01:44:56.762934 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.762899    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da800208-fbd5-460f-b1cc-a212a4445291-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.763577 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.763542    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnpvm\" (UniqueName: \"kubernetes.io/projected/da800208-fbd5-460f-b1cc-a212a4445291-kube-api-access-rnpvm\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.930522 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.929752    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:56.942306 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:56.942277    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64bf97a5-t6vfn"
Oct 31 01:44:57.287259 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:57.287229    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"]
Oct 31 01:44:57.289902 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:57.289863    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:44:57.289979 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:57.289946    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:44:57.290020 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:57.289988    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:44:57.291667 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:57.291638    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64bf97a5-t6vfn"]
Oct 31 01:44:57.348008 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:57.339546    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 01:44:57.348008 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:57.339608    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5" podUID="07620cf8-e211-41cf-8faf-177eeb1e2cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 01:44:57.440646 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:57.440609    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64bf97a5-t6vfn" event={"ID":"683dbaa0-6bdd-4431-8b3b-6ca550a9f509","Type":"ContainerStarted","Data":"a9b8d7868bae55895c98808821a77bdb7bf563d90bcccca55150c9d6ba6be6e0"}
Oct 31 01:44:57.442131 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:57.442100    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh" event={"ID":"da800208-fbd5-460f-b1cc-a212a4445291","Type":"ContainerStarted","Data":"bc929d49a9b4a1f807e39953fb2d7ed2e8d7e63a2e750f77fb486fa1b02830d2"}
Oct 31 01:44:58.324469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:58.323862    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 01:44:58.324469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:58.323940    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5" podUID="07620cf8-e211-41cf-8faf-177eeb1e2cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 01:44:58.460000 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:58.459956    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh" event={"ID":"da800208-fbd5-460f-b1cc-a212a4445291","Type":"ContainerStarted","Data":"43b83ebd4624f35d9ba435a41f1d41d2f23b44cd8c5dce683dc9aeca5732f362"}
Oct 31 01:44:58.461967 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:58.461935    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64bf97a5-t6vfn" event={"ID":"683dbaa0-6bdd-4431-8b3b-6ca550a9f509","Type":"ContainerStarted","Data":"7caafb5aa8ee71535cb85a9b8101d9056491496f8095cb29f3824bee2e190e8f"}
Oct 31 01:44:58.492084 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:58.492028    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh" podStartSLOduration=4.492013832 podStartE2EDuration="4.492013832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:58.486959779 +0000 UTC m=+1454.490674072" watchObservedRunningTime="2026-10-31 01:44:58.492013832 +0000 UTC m=+1454.495728123"
Oct 31 01:44:58.557163 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:58.557084    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64bf97a5-t6vfn" podStartSLOduration=3.557060141 podStartE2EDuration="3.557060141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:58.521261531 +0000 UTC m=+1454.524975823" watchObservedRunningTime="2026-10-31 01:44:58.557060141 +0000 UTC m=+1454.560774433"
Oct 31 01:44:58.590962 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:44:58.590871    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:44:58.935529 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:58.934725    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:44:58.935529 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:58.934902    5650 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 01:44:58.935529 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:58.934963    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh" podUID="da800208-fbd5-460f-b1cc-a212a4445291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 01:44:59.321470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:59.319874    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 01:44:59.321470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:59.319976    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5" podUID="07620cf8-e211-41cf-8faf-177eeb1e2cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 01:44:59.471199 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:59.471167    5650 generic.go:356] "Generic (PLEG): container finished" podID="683dbaa0-6bdd-4431-8b3b-6ca550a9f509" containerID="7caafb5aa8ee71535cb85a9b8101d9056491496f8095cb29f3824bee2e190e8f" exitCode=0
Oct 31 01:44:59.471642 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:59.471250    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64bf97a5-t6vfn" event={"ID":"683dbaa0-6bdd-4431-8b3b-6ca550a9f509","Type":"ContainerDied","Data":"7caafb5aa8ee71535cb85a9b8101d9056491496f8095cb29f3824bee2e190e8f"}
Oct 31 01:44:59.937469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:59.937303    5650 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 01:44:59.937469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:44:59.937374    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh" podUID="da800208-fbd5-460f-b1cc-a212a4445291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 01:45:00.153786 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.153751    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h"]
Oct 31 01:45:00.158620 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.158590    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn"]
Oct 31 01:45:00.158772 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.158693    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h"
Oct 31 01:45:00.164797 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.164489    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn"
Oct 31 01:45:00.171234 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.168699    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h"]
Oct 31 01:45:00.258465 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.250596    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/992f6f84-a9c6-475c-b0ca-17d24723014a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h\" (UID: \"992f6f84-a9c6-475c-b0ca-17d24723014a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h"
Oct 31 01:45:00.258465 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.250773    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgx4v\" (UniqueName: \"kubernetes.io/projected/1443bcfa-a35a-44fd-9aa8-893293e5522a-kube-api-access-zgx4v\") pod \"maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn\" (UID: \"1443bcfa-a35a-44fd-9aa8-893293e5522a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn"
Oct 31 01:45:00.258465 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.250829    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv86p\" (UniqueName: \"kubernetes.io/projected/992f6f84-a9c6-475c-b0ca-17d24723014a-kube-api-access-mv86p\") pod \"maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h\" (UID: \"992f6f84-a9c6-475c-b0ca-17d24723014a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h"
Oct 31 01:45:00.258465 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.250895    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1443bcfa-a35a-44fd-9aa8-893293e5522a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn\" (UID: \"1443bcfa-a35a-44fd-9aa8-893293e5522a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn"
Oct 31 01:45:00.279430 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.268199    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn"]
Oct 31 01:45:00.331585 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.327569    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 01:45:00.331585 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.327630    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5" podUID="07620cf8-e211-41cf-8faf-177eeb1e2cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 01:45:00.353189 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.352115    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/992f6f84-a9c6-475c-b0ca-17d24723014a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h\" (UID: \"992f6f84-a9c6-475c-b0ca-17d24723014a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h"
Oct 31 01:45:00.353189 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.352234    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgx4v\" (UniqueName: \"kubernetes.io/projected/1443bcfa-a35a-44fd-9aa8-893293e5522a-kube-api-access-zgx4v\") pod \"maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn\" (UID: \"1443bcfa-a35a-44fd-9aa8-893293e5522a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn"
Oct 31 01:45:00.353189 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.352265    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv86p\" (UniqueName: \"kubernetes.io/projected/992f6f84-a9c6-475c-b0ca-17d24723014a-kube-api-access-mv86p\") pod \"maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h\" (UID: \"992f6f84-a9c6-475c-b0ca-17d24723014a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h"
Oct 31 01:45:00.353189 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.352305    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1443bcfa-a35a-44fd-9aa8-893293e5522a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn\" (UID: \"1443bcfa-a35a-44fd-9aa8-893293e5522a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn"
Oct 31 01:45:00.353189 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.352965    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1443bcfa-a35a-44fd-9aa8-893293e5522a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn\" (UID: \"1443bcfa-a35a-44fd-9aa8-893293e5522a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn"
Oct 31 01:45:00.353590 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.353565    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/992f6f84-a9c6-475c-b0ca-17d24723014a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h\" (UID: \"992f6f84-a9c6-475c-b0ca-17d24723014a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h"
Oct 31 01:45:00.366393 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.366355    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgx4v\" (UniqueName: \"kubernetes.io/projected/1443bcfa-a35a-44fd-9aa8-893293e5522a-kube-api-access-zgx4v\") pod \"maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn\" (UID: \"1443bcfa-a35a-44fd-9aa8-893293e5522a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn"
Oct 31 01:45:00.366568 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.366393    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv86p\" (UniqueName: \"kubernetes.io/projected/992f6f84-a9c6-475c-b0ca-17d24723014a-kube-api-access-mv86p\") pod \"maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h\" (UID: \"992f6f84-a9c6-475c-b0ca-17d24723014a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h"
Oct 31 01:45:00.578357 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.578322    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h"
Oct 31 01:45:00.587988 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.587853    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn"
Oct 31 01:45:00.742129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.742095    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64bf97a5-t6vfn"
Oct 31 01:45:00.763109 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.762329    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/683dbaa0-6bdd-4431-8b3b-6ca550a9f509-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/683dbaa0-6bdd-4431-8b3b-6ca550a9f509-maas-api-openshift-ca\") pod \"683dbaa0-6bdd-4431-8b3b-6ca550a9f509\" (UID: \"683dbaa0-6bdd-4431-8b3b-6ca550a9f509\") "
Oct 31 01:45:00.772908 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.768401    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/683dbaa0-6bdd-4431-8b3b-6ca550a9f509-maas-api-openshift-ca" pod "683dbaa0-6bdd-4431-8b3b-6ca550a9f509" (UID: "683dbaa0-6bdd-4431-8b3b-6ca550a9f509"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:45:00.772908 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.768750    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/683dbaa0-6bdd-4431-8b3b-6ca550a9f509-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/683dbaa0-6bdd-4431-8b3b-6ca550a9f509-maas-api-xks-ca\") pod \"683dbaa0-6bdd-4431-8b3b-6ca550a9f509\" (UID: \"683dbaa0-6bdd-4431-8b3b-6ca550a9f509\") "
Oct 31 01:45:00.772908 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.770177    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/683dbaa0-6bdd-4431-8b3b-6ca550a9f509-kube-api-access-tqmrv\" (UniqueName: \"kubernetes.io/projected/683dbaa0-6bdd-4431-8b3b-6ca550a9f509-kube-api-access-tqmrv\") pod \"683dbaa0-6bdd-4431-8b3b-6ca550a9f509\" (UID: \"683dbaa0-6bdd-4431-8b3b-6ca550a9f509\") "
Oct 31 01:45:00.774194 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.774145    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/683dbaa0-6bdd-4431-8b3b-6ca550a9f509-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:00.843869 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.843755    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/683dbaa0-6bdd-4431-8b3b-6ca550a9f509-maas-api-xks-ca" pod "683dbaa0-6bdd-4431-8b3b-6ca550a9f509" (UID: "683dbaa0-6bdd-4431-8b3b-6ca550a9f509"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:45:00.871776 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.860706    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/683dbaa0-6bdd-4431-8b3b-6ca550a9f509-kube-api-access-tqmrv" pod "683dbaa0-6bdd-4431-8b3b-6ca550a9f509" (UID: "683dbaa0-6bdd-4431-8b3b-6ca550a9f509"). InnerVolumeSpecName "kube-api-access-tqmrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:45:00.880962 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.880901    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/683dbaa0-6bdd-4431-8b3b-6ca550a9f509-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:00.880962 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.880937    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqmrv\" (UniqueName: \"kubernetes.io/projected/683dbaa0-6bdd-4431-8b3b-6ca550a9f509-kube-api-access-tqmrv\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:00.935965 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.935916    5650 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 01:45:00.936150 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.935992    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh" podUID="da800208-fbd5-460f-b1cc-a212a4445291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 01:45:00.988883 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:00.988846    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h"]
Oct 31 01:45:00.991656 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:45:00.990950    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992f6f84_a9c6_475c_b0ca_17d24723014a.slice/crio-bb5dc7cc54899b8058249353a15a3eadf6d8161fcb737861eaf361d39504db68 WatchSource:0}: Error finding container bb5dc7cc54899b8058249353a15a3eadf6d8161fcb737861eaf361d39504db68: Status 404 returned error can't find the container with id bb5dc7cc54899b8058249353a15a3eadf6d8161fcb737861eaf361d39504db68
Oct 31 01:45:01.084737 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:45:01.084709    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1443bcfa_a35a_44fd_9aa8_893293e5522a.slice/crio-2cecfe52ee79aeca6b50be4d4c07a9e28b90e94cca4d27b60112145c97084a74 WatchSource:0}: Error finding container 2cecfe52ee79aeca6b50be4d4c07a9e28b90e94cca4d27b60112145c97084a74: Status 404 returned error can't find the container with id 2cecfe52ee79aeca6b50be4d4c07a9e28b90e94cca4d27b60112145c97084a74
Oct 31 01:45:01.085194 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:01.085170    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn"]
Oct 31 01:45:01.491184 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:01.491151    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64bf97a5-t6vfn"
Oct 31 01:45:01.491184 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:01.491171    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64bf97a5-t6vfn" event={"ID":"683dbaa0-6bdd-4431-8b3b-6ca550a9f509","Type":"ContainerDied","Data":"a9b8d7868bae55895c98808821a77bdb7bf563d90bcccca55150c9d6ba6be6e0"}
Oct 31 01:45:01.491459 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:01.491234    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9b8d7868bae55895c98808821a77bdb7bf563d90bcccca55150c9d6ba6be6e0"
Oct 31 01:45:01.509917 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:01.495948    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h" event={"ID":"992f6f84-a9c6-475c-b0ca-17d24723014a","Type":"ContainerStarted","Data":"bb5dc7cc54899b8058249353a15a3eadf6d8161fcb737861eaf361d39504db68"}
Oct 31 01:45:01.509917 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:01.498044    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn" event={"ID":"1443bcfa-a35a-44fd-9aa8-893293e5522a","Type":"ContainerStarted","Data":"2cecfe52ee79aeca6b50be4d4c07a9e28b90e94cca4d27b60112145c97084a74"}
Oct 31 01:45:01.867239 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:01.867208    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:45:01.867749 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:01.867652    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:45:01.868585 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:01.868561    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:45:01.932634 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:01.932494    5650 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 01:45:01.932634 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:01.932556    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh" podUID="da800208-fbd5-460f-b1cc-a212a4445291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 01:45:02.527310 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:02.527219    5650 generic.go:356] "Generic (PLEG): container finished" podID="1443bcfa-a35a-44fd-9aa8-893293e5522a" containerID="17a6ae413645fdd6d50533b86c300a4578f69f9c3d200b43b959efddc3d6f791" exitCode=60
Oct 31 01:45:02.528067 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:02.527760    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn" event={"ID":"1443bcfa-a35a-44fd-9aa8-893293e5522a","Type":"ContainerDied","Data":"17a6ae413645fdd6d50533b86c300a4578f69f9c3d200b43b959efddc3d6f791"}
Oct 31 01:45:02.528067 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:02.527913    5650 scope.go:122] "RemoveContainer" containerID="17a6ae413645fdd6d50533b86c300a4578f69f9c3d200b43b959efddc3d6f791"
Oct 31 01:45:02.533667 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:02.533534    5650 generic.go:356] "Generic (PLEG): container finished" podID="992f6f84-a9c6-475c-b0ca-17d24723014a" containerID="8d17c772d27aaa2c2ceaa22df20ff8917c5df85e86e3e90a0e261cde3df7ef95" exitCode=60
Oct 31 01:45:02.542041 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:02.536260    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h" event={"ID":"992f6f84-a9c6-475c-b0ca-17d24723014a","Type":"ContainerDied","Data":"8d17c772d27aaa2c2ceaa22df20ff8917c5df85e86e3e90a0e261cde3df7ef95"}
Oct 31 01:45:02.542041 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:02.536666    5650 scope.go:122] "RemoveContainer" containerID="8d17c772d27aaa2c2ceaa22df20ff8917c5df85e86e3e90a0e261cde3df7ef95"
Oct 31 01:45:02.930889 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:02.930853    5650 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 01:45:02.932082 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:02.932047    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh" podUID="da800208-fbd5-460f-b1cc-a212a4445291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 01:45:03.544231 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:03.544189    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h" event={"ID":"992f6f84-a9c6-475c-b0ca-17d24723014a","Type":"ContainerStarted","Data":"c06f46e67ec334e1e4feca0f7ccb2c5a7b405958062e858f6a9c3584af108069"}
Oct 31 01:45:03.546357 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:03.546319    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn" event={"ID":"1443bcfa-a35a-44fd-9aa8-893293e5522a","Type":"ContainerStarted","Data":"9debfb93dc01d398dbaef61e93a39c9168218252c492f7dc3f48fde7208fe5ba"}
Oct 31 01:45:03.564958 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:03.564900    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h" podStartSLOduration=3.564885216 podStartE2EDuration="3.564885216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:03.563789883 +0000 UTC m=+1459.567504174" watchObservedRunningTime="2026-10-31 01:45:03.564885216 +0000 UTC m=+1459.568599507"
Oct 31 01:45:03.586567 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:03.586515    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn" podStartSLOduration=3.586500305 podStartE2EDuration="3.586500305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:03.584109431 +0000 UTC m=+1459.587823787" watchObservedRunningTime="2026-10-31 01:45:03.586500305 +0000 UTC m=+1459.590214595"
Oct 31 01:45:03.931126 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:03.931088    5650 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 01:45:03.931553 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:03.931144    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh" podUID="da800208-fbd5-460f-b1cc-a212a4445291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 01:45:04.556296 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:04.556256    5650 generic.go:356] "Generic (PLEG): container finished" podID="992f6f84-a9c6-475c-b0ca-17d24723014a" containerID="c06f46e67ec334e1e4feca0f7ccb2c5a7b405958062e858f6a9c3584af108069" exitCode=60
Oct 31 01:45:04.556656 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:04.556349    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h" event={"ID":"992f6f84-a9c6-475c-b0ca-17d24723014a","Type":"ContainerDied","Data":"c06f46e67ec334e1e4feca0f7ccb2c5a7b405958062e858f6a9c3584af108069"}
Oct 31 01:45:04.556656 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:04.556427    5650 scope.go:122] "RemoveContainer" containerID="8d17c772d27aaa2c2ceaa22df20ff8917c5df85e86e3e90a0e261cde3df7ef95"
Oct 31 01:45:04.556796 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:04.556778    5650 scope.go:122] "RemoveContainer" containerID="c06f46e67ec334e1e4feca0f7ccb2c5a7b405958062e858f6a9c3584af108069"
Oct 31 01:45:04.557174 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:45:04.557144    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h_odh-ai-gateway-infra(992f6f84-a9c6-475c-b0ca-17d24723014a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h" podUID="992f6f84-a9c6-475c-b0ca-17d24723014a"
Oct 31 01:45:04.560428 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:04.560398    5650 generic.go:356] "Generic (PLEG): container finished" podID="1443bcfa-a35a-44fd-9aa8-893293e5522a" containerID="9debfb93dc01d398dbaef61e93a39c9168218252c492f7dc3f48fde7208fe5ba" exitCode=60
Oct 31 01:45:04.560601 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:04.560488    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn" event={"ID":"1443bcfa-a35a-44fd-9aa8-893293e5522a","Type":"ContainerDied","Data":"9debfb93dc01d398dbaef61e93a39c9168218252c492f7dc3f48fde7208fe5ba"}
Oct 31 01:45:04.560922 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:04.560901    5650 scope.go:122] "RemoveContainer" containerID="9debfb93dc01d398dbaef61e93a39c9168218252c492f7dc3f48fde7208fe5ba"
Oct 31 01:45:04.561830 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:45:04.561801    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn_odh-ai-gateway-infra(1443bcfa-a35a-44fd-9aa8-893293e5522a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn" podUID="1443bcfa-a35a-44fd-9aa8-893293e5522a"
Oct 31 01:45:04.577044 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:04.577015    5650 scope.go:122] "RemoveContainer" containerID="17a6ae413645fdd6d50533b86c300a4578f69f9c3d200b43b959efddc3d6f791"
Oct 31 01:45:05.019230 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:05.019193    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:45:05.021430 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:05.021406    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:45:05.027826 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:05.027800    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 01:45:05.576872 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:05.576833    5650 scope.go:122] "RemoveContainer" containerID="9debfb93dc01d398dbaef61e93a39c9168218252c492f7dc3f48fde7208fe5ba"
Oct 31 01:45:05.577601 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:45:05.577553    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn_odh-ai-gateway-infra(1443bcfa-a35a-44fd-9aa8-893293e5522a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn" podUID="1443bcfa-a35a-44fd-9aa8-893293e5522a"
Oct 31 01:45:05.580809 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:05.580787    5650 scope.go:122] "RemoveContainer" containerID="c06f46e67ec334e1e4feca0f7ccb2c5a7b405958062e858f6a9c3584af108069"
Oct 31 01:45:05.581074 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:45:05.581044    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h_odh-ai-gateway-infra(992f6f84-a9c6-475c-b0ca-17d24723014a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h" podUID="992f6f84-a9c6-475c-b0ca-17d24723014a"
Oct 31 01:45:05.899176 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:05.899070    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44dafe43-m46n5"]
Oct 31 01:45:05.901658 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:05.901622    5650 cpu_manager.go:414] "containerMap: removing container" podUID="683dbaa0-6bdd-4431-8b3b-6ca550a9f509" containerName="revoke-keys"
Oct 31 01:45:05.901658 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:05.901654    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="683dbaa0-6bdd-4431-8b3b-6ca550a9f509" containerName="revoke-keys"
Oct 31 01:45:05.901942 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:05.901927    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="683dbaa0-6bdd-4431-8b3b-6ca550a9f509" containerName="revoke-keys"
Oct 31 01:45:05.908898 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:05.908858    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dafe43-m46n5"
Oct 31 01:45:05.916204 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:05.916168    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44dafe43-m46n5"]
Oct 31 01:45:05.919295 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:05.917556    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:45:06.061003 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:06.056398    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg66j\" (UniqueName: \"kubernetes.io/projected/302ba23c-5ce9-433c-b423-8c4f5ee70d0f-kube-api-access-mg66j\") pod \"maas-api-revoke-sub-44dafe43-m46n5\" (UID: \"302ba23c-5ce9-433c-b423-8c4f5ee70d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dafe43-m46n5"
Oct 31 01:45:06.061003 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:06.056719    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302ba23c-5ce9-433c-b423-8c4f5ee70d0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44dafe43-m46n5\" (UID: \"302ba23c-5ce9-433c-b423-8c4f5ee70d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dafe43-m46n5"
Oct 31 01:45:06.061003 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:06.056773    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302ba23c-5ce9-433c-b423-8c4f5ee70d0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44dafe43-m46n5\" (UID: \"302ba23c-5ce9-433c-b423-8c4f5ee70d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dafe43-m46n5"
Oct 31 01:45:06.170663 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:06.168644    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302ba23c-5ce9-433c-b423-8c4f5ee70d0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44dafe43-m46n5\" (UID: \"302ba23c-5ce9-433c-b423-8c4f5ee70d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dafe43-m46n5"
Oct 31 01:45:06.170663 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:06.168721    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302ba23c-5ce9-433c-b423-8c4f5ee70d0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44dafe43-m46n5\" (UID: \"302ba23c-5ce9-433c-b423-8c4f5ee70d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dafe43-m46n5"
Oct 31 01:45:06.170663 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:06.168878    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg66j\" (UniqueName: \"kubernetes.io/projected/302ba23c-5ce9-433c-b423-8c4f5ee70d0f-kube-api-access-mg66j\") pod \"maas-api-revoke-sub-44dafe43-m46n5\" (UID: \"302ba23c-5ce9-433c-b423-8c4f5ee70d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dafe43-m46n5"
Oct 31 01:45:06.170663 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:06.170197    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302ba23c-5ce9-433c-b423-8c4f5ee70d0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44dafe43-m46n5\" (UID: \"302ba23c-5ce9-433c-b423-8c4f5ee70d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dafe43-m46n5"
Oct 31 01:45:06.206483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:06.200261    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302ba23c-5ce9-433c-b423-8c4f5ee70d0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44dafe43-m46n5\" (UID: \"302ba23c-5ce9-433c-b423-8c4f5ee70d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dafe43-m46n5"
Oct 31 01:45:06.225267 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:06.225231    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg66j\" (UniqueName: \"kubernetes.io/projected/302ba23c-5ce9-433c-b423-8c4f5ee70d0f-kube-api-access-mg66j\") pod \"maas-api-revoke-sub-44dafe43-m46n5\" (UID: \"302ba23c-5ce9-433c-b423-8c4f5ee70d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dafe43-m46n5"
Oct 31 01:45:06.239070 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:06.238783    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dafe43-m46n5"
Oct 31 01:45:07.150996 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:07.150473    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44dafe43-m46n5"]
Oct 31 01:45:07.736038 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:07.735991    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dafe43-m46n5" event={"ID":"302ba23c-5ce9-433c-b423-8c4f5ee70d0f","Type":"ContainerStarted","Data":"31a59f20490dfb093359b275373683d081e6d99f6142a0773c048b98da60bf42"}
Oct 31 01:45:07.799926 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:07.798555    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:07.854608 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:07.854543    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 01:45:07.935710 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:07.935660    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:07.955543 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:07.955440    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 31 01:45:07.996472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:07.993663    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:07.996472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:07.994846    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 01:45:08.035484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.034607    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:45:08.035484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.034676    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8" podUID="da0a52af-42a1-45d5-ab84-6696241d5489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:08.139537 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.137677    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:08.266469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.258640    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:08.358508 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.348801    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"]
Oct 31 01:45:08.513187 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.513096    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"]
Oct 31 01:45:08.513544 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.513521    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:08.530858 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.530647    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-cb5756cb-openshift-default-dockercfg-twxqj\""
Oct 31 01:45:08.618249 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.618204    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-workload-certs\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:08.618561 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.618534    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-workload-socket\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:08.618690 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.618673    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-token\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:08.618795 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.618782    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1357b02f-b722-40ac-8ec2-35f5626d8347-istiod-ca-cert\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:08.618889 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.618874    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-data\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:08.619063 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.619046    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92f6d\" (UniqueName: \"kubernetes.io/projected/1357b02f-b722-40ac-8ec2-35f5626d8347-kube-api-access-92f6d\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:08.619195 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.619179    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-envoy\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:08.619383 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.619363    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-credential-socket\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:08.619544 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.619523    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-podinfo\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:08.689471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.680607    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:08.732473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.720649    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-workload-certs\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:08.805196 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.805102    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-workload-socket\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:08.805416 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.751543    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:08.805936 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.724077    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-workload-certs\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:08.806123 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.806100    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-workload-socket\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:08.806271 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.806254    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-token\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:08.806386 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.806371    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1357b02f-b722-40ac-8ec2-35f5626d8347-istiod-ca-cert\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:08.806524 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.806508    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-data\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:08.855469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.835699    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92f6d\" (UniqueName: \"kubernetes.io/projected/1357b02f-b722-40ac-8ec2-35f5626d8347-kube-api-access-92f6d\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:08.855469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.835778    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-envoy\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:08.855469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.835893    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-credential-socket\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:08.855469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.835936    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-podinfo\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:08.855469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.813744    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:08.855469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.813382    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-data\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:08.855469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.840591    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:45:08.855469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.814309    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1357b02f-b722-40ac-8ec2-35f5626d8347-istiod-ca-cert\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:08.934488 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.931942    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-credential-socket\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:08.934488 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:08.933430    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 01:45:09.039407 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:09.038602    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:09.039407 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:09.039078    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 01:45:09.154693 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:09.149582    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 01:45:09.154693 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:09.152326    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:45:09.154693 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:09.152424    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:45:09.154693 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:09.152503    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:45:09.154693 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:09.152568    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:45:09.335433 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:09.324640    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:09.335433 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:09.327841    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:45:09.356372 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:09.356311    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 31 01:45:09.404897 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:09.401587    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:45:09.953640 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:09.953581    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 01:45:10.054873 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:10.050482    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:10.108488 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:10.089476    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:10.108488 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:10.090094    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Oct 31 01:45:10.108488 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:10.108266    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92f6d\" (UniqueName: \"kubernetes.io/projected/1357b02f-b722-40ac-8ec2-35f5626d8347-kube-api-access-92f6d\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:10.180493 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:10.153565    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-token\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:10.180493 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:10.154353    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-podinfo\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:10.181780 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:10.181655    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:10.263529 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:10.251764    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-envoy\") pod \"e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") " pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:10.276180 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:10.274688    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 01:45:10.299895 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:10.299813    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:10.327554 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:10.327378    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:10.357270 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:10.356755    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:10.554396 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:10.554284    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:10.596756 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:10.595722    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:45:10.598121 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:10.598089    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dafe43-m46n5" event={"ID":"302ba23c-5ce9-433c-b423-8c4f5ee70d0f","Type":"ContainerStarted","Data":"9ef4f1cfb18482bfc29e1cdf32fb054ddffd64bbac8280dfcf8b3e5fc15840c2"}
Oct 31 01:45:10.598315 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:10.598180    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:45:10.646790 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:10.645990    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:45:10.665979 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:10.653101    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:45:10.665979 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:10.659801    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:45:10.701016 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:10.700980    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:45:10.707426 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:10.702528    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:45:10.707426 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:10.703737    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:45:10.707426 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:10.703788    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:45:10.768728 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:10.768648    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dafe43-m46n5" podStartSLOduration=5.768624989 podStartE2EDuration="5.768624989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:10.761264942 +0000 UTC m=+1466.764979234" watchObservedRunningTime="2026-10-31 01:45:10.768624989 +0000 UTC m=+1466.772339280"
Oct 31 01:45:11.013065 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:11.013034    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"]
Oct 31 01:45:11.014699 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:45:11.014671    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1357b02f_b722_40ac_8ec2_35f5626d8347.slice/crio-e4bf28fcb669a2595cb706a558e89e3efa6c6724255a200b367e837475e27497 WatchSource:0}: Error finding container e4bf28fcb669a2595cb706a558e89e3efa6c6724255a200b367e837475e27497: Status 404 returned error can't find the container with id e4bf28fcb669a2595cb706a558e89e3efa6c6724255a200b367e837475e27497
Oct 31 01:45:11.017905 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:11.017872    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:45:11.017998 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:11.017950    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:45:11.018066 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:11.017993    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:45:11.234734 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:11.234686    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd" event={"ID":"1357b02f-b722-40ac-8ec2-35f5626d8347","Type":"ContainerStarted","Data":"97cb4f814576543ee642a0a0a5995e2c4c11197cd4b9e66d10192b234e51f864"}
Oct 31 01:45:11.234971 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:11.234742    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd" event={"ID":"1357b02f-b722-40ac-8ec2-35f5626d8347","Type":"ContainerStarted","Data":"e4bf28fcb669a2595cb706a558e89e3efa6c6724255a200b367e837475e27497"}
Oct 31 01:45:11.238433 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:11.238397    5650 generic.go:356] "Generic (PLEG): container finished" podID="302ba23c-5ce9-433c-b423-8c4f5ee70d0f" containerID="9ef4f1cfb18482bfc29e1cdf32fb054ddffd64bbac8280dfcf8b3e5fc15840c2" exitCode=0
Oct 31 01:45:11.238644 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:11.238533    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dafe43-m46n5" event={"ID":"302ba23c-5ce9-433c-b423-8c4f5ee70d0f","Type":"ContainerDied","Data":"9ef4f1cfb18482bfc29e1cdf32fb054ddffd64bbac8280dfcf8b3e5fc15840c2"}
Oct 31 01:45:11.276718 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:11.276602    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd" podStartSLOduration=3.276563112 podStartE2EDuration="3.276563112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:11.269982608 +0000 UTC m=+1467.273696900" watchObservedRunningTime="2026-10-31 01:45:11.276563112 +0000 UTC m=+1467.280277404"
Oct 31 01:45:11.560848 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:11.560756    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:12.427515 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:12.427477    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dafe43-m46n5"
Oct 31 01:45:12.452763 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:12.452728    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/302ba23c-5ce9-433c-b423-8c4f5ee70d0f-kube-api-access-mg66j\" (UniqueName: \"kubernetes.io/projected/302ba23c-5ce9-433c-b423-8c4f5ee70d0f-kube-api-access-mg66j\") pod \"302ba23c-5ce9-433c-b423-8c4f5ee70d0f\" (UID: \"302ba23c-5ce9-433c-b423-8c4f5ee70d0f\") "
Oct 31 01:45:12.452960 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:12.452939    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/302ba23c-5ce9-433c-b423-8c4f5ee70d0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302ba23c-5ce9-433c-b423-8c4f5ee70d0f-maas-api-xks-ca\") pod \"302ba23c-5ce9-433c-b423-8c4f5ee70d0f\" (UID: \"302ba23c-5ce9-433c-b423-8c4f5ee70d0f\") "
Oct 31 01:45:12.453033 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:12.453015    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/302ba23c-5ce9-433c-b423-8c4f5ee70d0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302ba23c-5ce9-433c-b423-8c4f5ee70d0f-maas-api-openshift-ca\") pod \"302ba23c-5ce9-433c-b423-8c4f5ee70d0f\" (UID: \"302ba23c-5ce9-433c-b423-8c4f5ee70d0f\") "
Oct 31 01:45:12.453779 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:12.453747    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/302ba23c-5ce9-433c-b423-8c4f5ee70d0f-maas-api-openshift-ca" pod "302ba23c-5ce9-433c-b423-8c4f5ee70d0f" (UID: "302ba23c-5ce9-433c-b423-8c4f5ee70d0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:45:12.458505 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:12.457541    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/302ba23c-5ce9-433c-b423-8c4f5ee70d0f-kube-api-access-mg66j" pod "302ba23c-5ce9-433c-b423-8c4f5ee70d0f" (UID: "302ba23c-5ce9-433c-b423-8c4f5ee70d0f"). InnerVolumeSpecName "kube-api-access-mg66j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:45:12.458505 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:12.457747    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/302ba23c-5ce9-433c-b423-8c4f5ee70d0f-maas-api-xks-ca" pod "302ba23c-5ce9-433c-b423-8c4f5ee70d0f" (UID: "302ba23c-5ce9-433c-b423-8c4f5ee70d0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:45:12.555362 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:12.555266    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302ba23c-5ce9-433c-b423-8c4f5ee70d0f-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:12.555362 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:12.555301    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302ba23c-5ce9-433c-b423-8c4f5ee70d0f-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:12.555362 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:12.555311    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg66j\" (UniqueName: \"kubernetes.io/projected/302ba23c-5ce9-433c-b423-8c4f5ee70d0f-kube-api-access-mg66j\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:12.566732 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:12.566688    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:13.256139 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:13.256106    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dafe43-m46n5"
Oct 31 01:45:13.256139 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:13.256130    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dafe43-m46n5" event={"ID":"302ba23c-5ce9-433c-b423-8c4f5ee70d0f","Type":"ContainerDied","Data":"31a59f20490dfb093359b275373683d081e6d99f6142a0773c048b98da60bf42"}
Oct 31 01:45:13.256407 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:13.256175    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31a59f20490dfb093359b275373683d081e6d99f6142a0773c048b98da60bf42"
Oct 31 01:45:13.256529 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:13.256510    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:13.257654 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:13.257630    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:45:14.963656 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:14.963618    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh"]
Oct 31 01:45:14.964710 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:14.964679    5650 cpu_manager.go:414] "containerMap: removing container" podUID="302ba23c-5ce9-433c-b423-8c4f5ee70d0f" containerName="revoke-keys"
Oct 31 01:45:14.964710 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:14.964712    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="302ba23c-5ce9-433c-b423-8c4f5ee70d0f" containerName="revoke-keys"
Oct 31 01:45:14.964904 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:14.964862    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="302ba23c-5ce9-433c-b423-8c4f5ee70d0f" containerName="revoke-keys"
Oct 31 01:45:14.969197 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:14.969172    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh"
Oct 31 01:45:14.979834 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:14.979807    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cb5756cb-dockercfg-5bpdg\""
Oct 31 01:45:15.002730 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:15.002692    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb"]
Oct 31 01:45:15.004838 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:15.004811    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-cb5756cb\""
Oct 31 01:45:15.009118 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:15.009078    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb"
Oct 31 01:45:15.009318 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:15.009218    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh"]
Oct 31 01:45:15.107469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:15.106542    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d359baf9-7e49-4e89-afb1-7c763722c087-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh\" (UID: \"d359baf9-7e49-4e89-afb1-7c763722c087\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh"
Oct 31 01:45:15.107469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:15.106639    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdt4k\" (UniqueName: \"kubernetes.io/projected/2946cec0-bd91-40ac-a095-6e174a3bf3b2-kube-api-access-vdt4k\") pod \"payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb\" (UID: \"2946cec0-bd91-40ac-a095-6e174a3bf3b2\") " pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb"
Oct 31 01:45:15.107469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:15.106739    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2946cec0-bd91-40ac-a095-6e174a3bf3b2-plugins-config-volume\") pod \"payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb\" (UID: \"2946cec0-bd91-40ac-a095-6e174a3bf3b2\") " pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb"
Oct 31 01:45:15.107469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:15.106764    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p7kd\" (UniqueName: \"kubernetes.io/projected/d359baf9-7e49-4e89-afb1-7c763722c087-kube-api-access-7p7kd\") pod \"payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh\" (UID: \"d359baf9-7e49-4e89-afb1-7c763722c087\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh"
Oct 31 01:45:15.148565 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:15.148524    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb"]
Oct 31 01:45:15.208040 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:15.207999    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2946cec0-bd91-40ac-a095-6e174a3bf3b2-plugins-config-volume\") pod \"payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb\" (UID: \"2946cec0-bd91-40ac-a095-6e174a3bf3b2\") " pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb"
Oct 31 01:45:15.208040 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:15.208041    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p7kd\" (UniqueName: \"kubernetes.io/projected/d359baf9-7e49-4e89-afb1-7c763722c087-kube-api-access-7p7kd\") pod \"payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh\" (UID: \"d359baf9-7e49-4e89-afb1-7c763722c087\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh"
Oct 31 01:45:15.208440 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:15.208422    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d359baf9-7e49-4e89-afb1-7c763722c087-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh\" (UID: \"d359baf9-7e49-4e89-afb1-7c763722c087\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh"
Oct 31 01:45:15.208553 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:15.208538    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdt4k\" (UniqueName: \"kubernetes.io/projected/2946cec0-bd91-40ac-a095-6e174a3bf3b2-kube-api-access-vdt4k\") pod \"payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb\" (UID: \"2946cec0-bd91-40ac-a095-6e174a3bf3b2\") " pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb"
Oct 31 01:45:15.209736 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:15.209713    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d359baf9-7e49-4e89-afb1-7c763722c087-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh\" (UID: \"d359baf9-7e49-4e89-afb1-7c763722c087\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh"
Oct 31 01:45:15.210215 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:15.210188    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2946cec0-bd91-40ac-a095-6e174a3bf3b2-plugins-config-volume\") pod \"payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb\" (UID: \"2946cec0-bd91-40ac-a095-6e174a3bf3b2\") " pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb"
Oct 31 01:45:15.223996 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:15.223912    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p7kd\" (UniqueName: \"kubernetes.io/projected/d359baf9-7e49-4e89-afb1-7c763722c087-kube-api-access-7p7kd\") pod \"payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh\" (UID: \"d359baf9-7e49-4e89-afb1-7c763722c087\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh"
Oct 31 01:45:15.231491 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:15.228236    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdt4k\" (UniqueName: \"kubernetes.io/projected/2946cec0-bd91-40ac-a095-6e174a3bf3b2-kube-api-access-vdt4k\") pod \"payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb\" (UID: \"2946cec0-bd91-40ac-a095-6e174a3bf3b2\") " pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb"
Oct 31 01:45:15.285497 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:15.284631    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh"
Oct 31 01:45:15.397265 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:15.397230    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb"
Oct 31 01:45:15.624642 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:15.624604    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh"]
Oct 31 01:45:15.634426 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:45:15.634388    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd359baf9_7e49_4e89_afb1_7c763722c087.slice/crio-37cf62a4dbba4e748c18236aa03edf611a4d35a7eb3de23f40d2e089b088fe52 WatchSource:0}: Error finding container 37cf62a4dbba4e748c18236aa03edf611a4d35a7eb3de23f40d2e089b088fe52: Status 404 returned error can't find the container with id 37cf62a4dbba4e748c18236aa03edf611a4d35a7eb3de23f40d2e089b088fe52
Oct 31 01:45:15.730300 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:15.730200    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb"]
Oct 31 01:45:16.110993 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:16.110913    5650 scope.go:122] "RemoveContainer" containerID="c06f46e67ec334e1e4feca0f7ccb2c5a7b405958062e858f6a9c3584af108069"
Oct 31 01:45:16.280777 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:16.280727    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb" event={"ID":"2946cec0-bd91-40ac-a095-6e174a3bf3b2","Type":"ContainerStarted","Data":"4ae41fe256e8d20d8b3da54e28e028ee2cc4a10f3db18521a6d35c26094abbbb"}
Oct 31 01:45:16.280941 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:16.280802    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb" event={"ID":"2946cec0-bd91-40ac-a095-6e174a3bf3b2","Type":"ContainerStarted","Data":"20cf91b1823e235eab7e6ef19b36b076bce7fad09b9e4e29b316ba82c8ad5fd6"}
Oct 31 01:45:16.280941 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:16.280923    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb"
Oct 31 01:45:16.282733 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:16.282698    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh" event={"ID":"d359baf9-7e49-4e89-afb1-7c763722c087","Type":"ContainerStarted","Data":"24137e3dc6c858268fb4332653f93731f7f851fd1bf525ed59f94359ddbc5c50"}
Oct 31 01:45:16.282883 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:16.282752    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh"
Oct 31 01:45:16.282883 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:16.282770    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh" event={"ID":"d359baf9-7e49-4e89-afb1-7c763722c087","Type":"ContainerStarted","Data":"37cf62a4dbba4e748c18236aa03edf611a4d35a7eb3de23f40d2e089b088fe52"}
Oct 31 01:45:16.300797 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:16.300743    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb" podStartSLOduration=2.300729162 podStartE2EDuration="2.300729162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:16.299878799 +0000 UTC m=+1472.303593091" watchObservedRunningTime="2026-10-31 01:45:16.300729162 +0000 UTC m=+1472.304443453"
Oct 31 01:45:16.324110 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:16.324058    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh" podStartSLOduration=2.324043609 podStartE2EDuration="2.324043609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:16.32002784 +0000 UTC m=+1472.323742130" watchObservedRunningTime="2026-10-31 01:45:16.324043609 +0000 UTC m=+1472.327757899"
Oct 31 01:45:17.129484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:17.115205    5650 scope.go:122] "RemoveContainer" containerID="9debfb93dc01d398dbaef61e93a39c9168218252c492f7dc3f48fde7208fe5ba"
Oct 31 01:45:17.293703 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:17.293600    5650 generic.go:356] "Generic (PLEG): container finished" podID="992f6f84-a9c6-475c-b0ca-17d24723014a" containerID="bf3b6ae4c22ad77727266782086dc1840fd6c0eb8dd777dc7d12876a8268a316" exitCode=60
Oct 31 01:45:17.294635 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:17.294608    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h" event={"ID":"992f6f84-a9c6-475c-b0ca-17d24723014a","Type":"ContainerDied","Data":"bf3b6ae4c22ad77727266782086dc1840fd6c0eb8dd777dc7d12876a8268a316"}
Oct 31 01:45:17.294759 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:17.294665    5650 scope.go:122] "RemoveContainer" containerID="c06f46e67ec334e1e4feca0f7ccb2c5a7b405958062e858f6a9c3584af108069"
Oct 31 01:45:17.295567 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:17.295552    5650 scope.go:122] "RemoveContainer" containerID="bf3b6ae4c22ad77727266782086dc1840fd6c0eb8dd777dc7d12876a8268a316"
Oct 31 01:45:17.295858 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:45:17.295834    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h_odh-ai-gateway-infra(992f6f84-a9c6-475c-b0ca-17d24723014a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h" podUID="992f6f84-a9c6-475c-b0ca-17d24723014a"
Oct 31 01:45:18.304612 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:18.304578    5650 generic.go:356] "Generic (PLEG): container finished" podID="1443bcfa-a35a-44fd-9aa8-893293e5522a" containerID="a2f0217143d0209f80cc3f7a8f185de0a498910e011c28be45c1f29d927bdd40" exitCode=60
Oct 31 01:45:18.305025 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:18.304656    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn" event={"ID":"1443bcfa-a35a-44fd-9aa8-893293e5522a","Type":"ContainerDied","Data":"a2f0217143d0209f80cc3f7a8f185de0a498910e011c28be45c1f29d927bdd40"}
Oct 31 01:45:18.305025 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:18.304718    5650 scope.go:122] "RemoveContainer" containerID="9debfb93dc01d398dbaef61e93a39c9168218252c492f7dc3f48fde7208fe5ba"
Oct 31 01:45:18.305165 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:18.305150    5650 scope.go:122] "RemoveContainer" containerID="a2f0217143d0209f80cc3f7a8f185de0a498910e011c28be45c1f29d927bdd40"
Oct 31 01:45:18.305548 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:45:18.305529    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn_odh-ai-gateway-infra(1443bcfa-a35a-44fd-9aa8-893293e5522a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn" podUID="1443bcfa-a35a-44fd-9aa8-893293e5522a"
Oct 31 01:45:18.330404 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:18.330372    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h"]
Oct 31 01:45:18.519057 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:18.517667    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h"
Oct 31 01:45:18.531476 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:18.527643    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/992f6f84-a9c6-475c-b0ca-17d24723014a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/992f6f84-a9c6-475c-b0ca-17d24723014a-maas-api-service-ca\") pod \"992f6f84-a9c6-475c-b0ca-17d24723014a\" (UID: \"992f6f84-a9c6-475c-b0ca-17d24723014a\") "
Oct 31 01:45:18.531476 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:18.527863    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/992f6f84-a9c6-475c-b0ca-17d24723014a-kube-api-access-mv86p\" (UniqueName: \"kubernetes.io/projected/992f6f84-a9c6-475c-b0ca-17d24723014a-kube-api-access-mv86p\") pod \"992f6f84-a9c6-475c-b0ca-17d24723014a\" (UID: \"992f6f84-a9c6-475c-b0ca-17d24723014a\") "
Oct 31 01:45:18.531476 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:18.529078    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/992f6f84-a9c6-475c-b0ca-17d24723014a-maas-api-service-ca" pod "992f6f84-a9c6-475c-b0ca-17d24723014a" (UID: "992f6f84-a9c6-475c-b0ca-17d24723014a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:45:18.535313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:18.535278    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992f6f84-a9c6-475c-b0ca-17d24723014a-kube-api-access-mv86p" pod "992f6f84-a9c6-475c-b0ca-17d24723014a" (UID: "992f6f84-a9c6-475c-b0ca-17d24723014a"). InnerVolumeSpecName "kube-api-access-mv86p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:45:18.632284 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:18.632148    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/992f6f84-a9c6-475c-b0ca-17d24723014a-maas-api-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:18.632284 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:18.632241    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv86p\" (UniqueName: \"kubernetes.io/projected/992f6f84-a9c6-475c-b0ca-17d24723014a-kube-api-access-mv86p\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:19.313882 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:19.313850    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h"
Oct 31 01:45:19.314360 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:19.313887    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h" event={"ID":"992f6f84-a9c6-475c-b0ca-17d24723014a","Type":"ContainerDied","Data":"bb5dc7cc54899b8058249353a15a3eadf6d8161fcb737861eaf361d39504db68"}
Oct 31 01:45:19.314360 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:19.313951    5650 scope.go:122] "RemoveContainer" containerID="bf3b6ae4c22ad77727266782086dc1840fd6c0eb8dd777dc7d12876a8268a316"
Oct 31 01:45:19.342734 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:19.342698    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn"]
Oct 31 01:45:19.346825 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:19.346794    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h"]
Oct 31 01:45:19.358891 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:19.358857    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0e8b5c-29890185-n2l7h"]
Oct 31 01:45:19.527988 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:19.527958    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn"
Oct 31 01:45:19.662237 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:19.662139    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1443bcfa-a35a-44fd-9aa8-893293e5522a-kube-api-access-zgx4v\" (UniqueName: \"kubernetes.io/projected/1443bcfa-a35a-44fd-9aa8-893293e5522a-kube-api-access-zgx4v\") pod \"1443bcfa-a35a-44fd-9aa8-893293e5522a\" (UID: \"1443bcfa-a35a-44fd-9aa8-893293e5522a\") "
Oct 31 01:45:19.662412 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:19.662248    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1443bcfa-a35a-44fd-9aa8-893293e5522a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1443bcfa-a35a-44fd-9aa8-893293e5522a-maas-api-service-ca\") pod \"1443bcfa-a35a-44fd-9aa8-893293e5522a\" (UID: \"1443bcfa-a35a-44fd-9aa8-893293e5522a\") "
Oct 31 01:45:19.663385 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:19.663355    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1443bcfa-a35a-44fd-9aa8-893293e5522a-maas-api-service-ca" pod "1443bcfa-a35a-44fd-9aa8-893293e5522a" (UID: "1443bcfa-a35a-44fd-9aa8-893293e5522a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:45:19.680325 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:19.680298    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1443bcfa-a35a-44fd-9aa8-893293e5522a-kube-api-access-zgx4v" pod "1443bcfa-a35a-44fd-9aa8-893293e5522a" (UID: "1443bcfa-a35a-44fd-9aa8-893293e5522a"). InnerVolumeSpecName "kube-api-access-zgx4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:45:19.765173 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:19.765140    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1443bcfa-a35a-44fd-9aa8-893293e5522a-maas-api-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:19.765360 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:19.765176    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgx4v\" (UniqueName: \"kubernetes.io/projected/1443bcfa-a35a-44fd-9aa8-893293e5522a-kube-api-access-zgx4v\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:45:20.119870 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:20.119837    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="992f6f84-a9c6-475c-b0ca-17d24723014a" path="/var/lib/kubelet/pods/992f6f84-a9c6-475c-b0ca-17d24723014a/volumes"
Oct 31 01:45:20.337559 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:20.337510    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn" event={"ID":"1443bcfa-a35a-44fd-9aa8-893293e5522a","Type":"ContainerDied","Data":"2cecfe52ee79aeca6b50be4d4c07a9e28b90e94cca4d27b60112145c97084a74"}
Oct 31 01:45:20.338048 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:20.337657    5650 scope.go:122] "RemoveContainer" containerID="a2f0217143d0209f80cc3f7a8f185de0a498910e011c28be45c1f29d927bdd40"
Oct 31 01:45:20.338048 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:20.337874    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn"
Oct 31 01:45:20.368396 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:20.368362    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn"]
Oct 31 01:45:20.374837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:20.374761    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890185-zbbjn"]
Oct 31 01:45:22.119226 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:22.119194    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1443bcfa-a35a-44fd-9aa8-893293e5522a" path="/var/lib/kubelet/pods/1443bcfa-a35a-44fd-9aa8-893293e5522a/volumes"
Oct 31 01:45:27.299364 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:27.299332    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb"
Oct 31 01:45:27.299883 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:27.299413    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh"
Oct 31 01:45:28.553029 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:45:28.552989    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:45:33.370471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:33.357978    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:33.377587 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:33.377535    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:33.673039 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:33.672929    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 01:45:33.718106 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:33.714882    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:33.718106 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:33.715340    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 01:45:33.718106 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:33.715658    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Oct 31 01:45:33.718106 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:33.717391    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:33.825875 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:33.825807    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 01:45:34.500185 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.498911    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 31 01:45:34.517474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.502575    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 01:45:34.607349 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.607311    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"]
Oct 31 01:45:34.608577 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.608549    5650 cpu_manager.go:414] "containerMap: removing container" podUID="992f6f84-a9c6-475c-b0ca-17d24723014a" containerName="cleanup"
Oct 31 01:45:34.608747 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.608732    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992f6f84-a9c6-475c-b0ca-17d24723014a" containerName="cleanup"
Oct 31 01:45:34.628303 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.628264    5650 cpu_manager.go:414] "containerMap: removing container" podUID="992f6f84-a9c6-475c-b0ca-17d24723014a" containerName="cleanup"
Oct 31 01:45:34.628569 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.628547    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992f6f84-a9c6-475c-b0ca-17d24723014a" containerName="cleanup"
Oct 31 01:45:34.628740 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.628727    5650 cpu_manager.go:414] "containerMap: removing container" podUID="1443bcfa-a35a-44fd-9aa8-893293e5522a" containerName="cleanup"
Oct 31 01:45:34.628825 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.628814    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1443bcfa-a35a-44fd-9aa8-893293e5522a" containerName="cleanup"
Oct 31 01:45:34.628928 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.628915    5650 cpu_manager.go:414] "containerMap: removing container" podUID="1443bcfa-a35a-44fd-9aa8-893293e5522a" containerName="cleanup"
Oct 31 01:45:34.629021 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.629008    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1443bcfa-a35a-44fd-9aa8-893293e5522a" containerName="cleanup"
Oct 31 01:45:34.629122 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.629099    5650 cpu_manager.go:414] "containerMap: removing container" podUID="1443bcfa-a35a-44fd-9aa8-893293e5522a" containerName="cleanup"
Oct 31 01:45:34.629220 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.629207    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1443bcfa-a35a-44fd-9aa8-893293e5522a" containerName="cleanup"
Oct 31 01:45:34.629579 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.629557    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="1443bcfa-a35a-44fd-9aa8-893293e5522a" containerName="cleanup"
Oct 31 01:45:34.629714 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.629700    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="992f6f84-a9c6-475c-b0ca-17d24723014a" containerName="cleanup"
Oct 31 01:45:34.629808 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.629797    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="1443bcfa-a35a-44fd-9aa8-893293e5522a" containerName="cleanup"
Oct 31 01:45:34.629920 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.629907    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="992f6f84-a9c6-475c-b0ca-17d24723014a" containerName="cleanup"
Oct 31 01:45:34.685073 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.676592    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 01:45:34.685073 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.677576    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:45:34.685073 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.677609    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"]
Oct 31 01:45:34.685073 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.678565    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:34.685073 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.679531    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 01:45:34.685073 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.679805    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:45:34.685073 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.682536    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:34.697059 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.697023    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1b634e-openshift-default-dockercfg-qdmlf\""
Oct 31 01:45:34.715980 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.715939    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Oct 31 01:45:34.718292 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.718255    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:34.719564 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.719435    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 01:45:34.765513 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.734388    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 01:45:34.765513 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.761489    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-istio-data\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:34.765513 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.761577    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80996937-b751-47a3-a515-0bef2d5e2327-istio-podinfo\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:34.765513 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.761607    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-credential-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:34.765513 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.761681    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-workload-certs\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:34.765513 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.761747    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80996937-b751-47a3-a515-0bef2d5e2327-istiod-ca-cert\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:34.765513 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.761776    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-istio-envoy\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:34.765513 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.761850    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80996937-b751-47a3-a515-0bef2d5e2327-istio-token\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:34.765513 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.761956    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlbqc\" (UniqueName: \"kubernetes.io/projected/80996937-b751-47a3-a515-0bef2d5e2327-kube-api-access-wlbqc\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:34.765513 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.762035    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-workload-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:34.830498 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.791604    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:45:34.830498 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.791686    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:45:34.830498 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.791730    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:45:34.830498 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.791765    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:45:34.881828 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.869304    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 01:45:34.881828 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.869541    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:45:34.909471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.882205    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlbqc\" (UniqueName: \"kubernetes.io/projected/80996937-b751-47a3-a515-0bef2d5e2327-kube-api-access-wlbqc\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:34.909471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.882327    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-workload-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:34.909471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.882541    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-istio-data\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:34.909471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.882645    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80996937-b751-47a3-a515-0bef2d5e2327-istio-podinfo\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:34.909471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.882688    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-credential-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:34.909471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.882780    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-workload-certs\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:34.909471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.882876    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80996937-b751-47a3-a515-0bef2d5e2327-istiod-ca-cert\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:34.909471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.882909    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-istio-envoy\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:34.909471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.883010    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80996937-b751-47a3-a515-0bef2d5e2327-istio-token\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:34.909471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.884286    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-workload-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:34.909471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.884367    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-credential-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:34.909471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.885270    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-istio-data\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:34.909471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.887310    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80996937-b751-47a3-a515-0bef2d5e2327-istiod-ca-cert\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:34.912694 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:34.912654    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-workload-certs\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:35.219525 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:35.219322    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-istio-envoy\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:35.258550 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:35.258476    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80996937-b751-47a3-a515-0bef2d5e2327-istio-podinfo\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:35.404000 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:35.346558    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:35.405064 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:35.404960    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlbqc\" (UniqueName: \"kubernetes.io/projected/80996937-b751-47a3-a515-0bef2d5e2327-kube-api-access-wlbqc\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:35.656599 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:35.528194    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:35.656599 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:35.528683    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:35.656599 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:35.529028    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:45:35.656599 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:35.536166    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80996937-b751-47a3-a515-0bef2d5e2327-istio-token\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:35.656599 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:35.544973    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:35.692918 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:35.692844    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 01:45:35.839871 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:35.758165    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:45:35.839871 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:35.759706    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:35.839871 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:35.759996    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:35.839871 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:35.760227    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:35.839871 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:35.762226    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 01:45:35.871877 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:35.863967    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:35.910231 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:35.909676    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 01:45:36.342714 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:36.333014    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-28qvk" podUID="1548a9c1-776d-41f4-addb-399ab46f60c0" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.24:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:36.353467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:36.353325    5650 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:45:36.357540 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:36.353743    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh" podUID="da800208-fbd5-460f-b1cc-a212a4445291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:36.585780 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:36.585723    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:36.586785 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:36.586747    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:36.777613 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:36.777555    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:36.782589 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:36.782547    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:36.782894 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:36.782851    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:36.783041 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:36.783007    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:36.797605 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:36.797555    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:36.825899 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:36.810505    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:36.882165 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:36.882113    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:36.922079 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:36.921625    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 01:45:37.254473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:37.252619    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:45:37.285472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:37.282579    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7" podUID="dedb244a-f01c-463e-8201-c6d0ad8e3f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:37.608473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:37.607848    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 31 01:45:37.608473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:37.608265    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 01:45:37.824590 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:37.786668    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 01:45:37.824590 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:37.804165    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:37.824590 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:37.817163    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:37.909648 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:37.909584    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:37.941313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:37.914109    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 01:45:37.941313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:37.914151    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:37.969321 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:37.926018    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 01:45:38.167158 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:38.152646    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded"
Oct 31 01:45:38.272467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:38.258495    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"]
Oct 31 01:45:38.287656 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:38.274630    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:45:38.287995 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:38.287953    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:45:38.294642 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:38.294598    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:45:38.613990 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:38.612105    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:38.614348 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:38.614289    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:38.751697 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:38.736342    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:38.789221 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:38.789171    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:38.817618 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:38.817563    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:38.818546 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:38.818508    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:38.860948 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:38.859996    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:45:38.935368 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:38.933606    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 01:45:38.935368 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:38.933996    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:38.935368 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:38.934611    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 01:45:38.979481 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:38.967812    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:39.467752 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:39.467702    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx" event={"ID":"80996937-b751-47a3-a515-0bef2d5e2327","Type":"ContainerStarted","Data":"41b39ca463e59e2824893463401265b428c4a594ab95f9540fc4ff24936e59f0"}
Oct 31 01:45:39.620569 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:39.620516    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:39.834699 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:39.799664    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:39.835027 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:39.810204    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:45:39.835203 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:39.835190    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:45:39.906521 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:39.902673    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 01:45:39.906521 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:39.906380    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:45:39.938520 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:39.921722    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:45:39.938520 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:39.934786    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 01:45:39.938520 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:39.937800    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:45:39.948493 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:39.945068    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:45:40.004931 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.004901    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:45:40.005459 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.005389    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:45:40.048467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.025461    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:45:40.201392 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.201276    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"]
Oct 31 01:45:40.202897 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.202871    5650 cpu_manager.go:414] "containerMap: removing container" podUID="992f6f84-a9c6-475c-b0ca-17d24723014a" containerName="cleanup"
Oct 31 01:45:40.203122 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.203106    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992f6f84-a9c6-475c-b0ca-17d24723014a" containerName="cleanup"
Oct 31 01:45:40.203508 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.203490    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="992f6f84-a9c6-475c-b0ca-17d24723014a" containerName="cleanup"
Oct 31 01:45:40.203669 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.203656    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="1443bcfa-a35a-44fd-9aa8-893293e5522a" containerName="cleanup"
Oct 31 01:45:40.223925 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.223895    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:45:40.237740 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.237602    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cb5756cb\"/\"kube-root-ca.crt\""
Oct 31 01:45:40.241127 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.238134    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cb5756cb\"/\"e2e-lifecycle-model-cb5756cb-kserve-self-signed-certs\""
Oct 31 01:45:40.241127 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.238248    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cb5756cb\"/\"default-dockercfg-fkrtd\""
Oct 31 01:45:40.241127 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.238325    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cb5756cb\"/\"openshift-service-ca.crt\""
Oct 31 01:45:40.285421 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.283169    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-tmp-dir\") pod \"e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55\" (UID: \"96bb0499-5edf-49b8-8391-f7948587ff19\") " pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:45:40.285421 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.283390    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-model-cache\") pod \"e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55\" (UID: \"96bb0499-5edf-49b8-8391-f7948587ff19\") " pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:45:40.285421 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.283601    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmk74\" (UniqueName: \"kubernetes.io/projected/96bb0499-5edf-49b8-8391-f7948587ff19-kube-api-access-jmk74\") pod \"e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55\" (UID: \"96bb0499-5edf-49b8-8391-f7948587ff19\") " pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:45:40.285421 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.283644    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-home\") pod \"e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55\" (UID: \"96bb0499-5edf-49b8-8391-f7948587ff19\") " pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:45:40.285421 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.283710    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96bb0499-5edf-49b8-8391-f7948587ff19-tls-certs\") pod \"e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55\" (UID: \"96bb0499-5edf-49b8-8391-f7948587ff19\") " pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:45:40.285421 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.283780    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-dshm\") pod \"e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55\" (UID: \"96bb0499-5edf-49b8-8391-f7948587ff19\") " pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:45:40.303440 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.303400    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"]
Oct 31 01:45:40.385675 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.384779    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-tmp-dir\") pod \"e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55\" (UID: \"96bb0499-5edf-49b8-8391-f7948587ff19\") " pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:45:40.385675 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.384941    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-model-cache\") pod \"e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55\" (UID: \"96bb0499-5edf-49b8-8391-f7948587ff19\") " pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:45:40.385675 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.384999    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmk74\" (UniqueName: \"kubernetes.io/projected/96bb0499-5edf-49b8-8391-f7948587ff19-kube-api-access-jmk74\") pod \"e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55\" (UID: \"96bb0499-5edf-49b8-8391-f7948587ff19\") " pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:45:40.385675 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.385031    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-home\") pod \"e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55\" (UID: \"96bb0499-5edf-49b8-8391-f7948587ff19\") " pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:45:40.385675 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.385079    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96bb0499-5edf-49b8-8391-f7948587ff19-tls-certs\") pod \"e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55\" (UID: \"96bb0499-5edf-49b8-8391-f7948587ff19\") " pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:45:40.385675 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.385146    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-dshm\") pod \"e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55\" (UID: \"96bb0499-5edf-49b8-8391-f7948587ff19\") " pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:45:40.388796 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.386784    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-tmp-dir\") pod \"e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55\" (UID: \"96bb0499-5edf-49b8-8391-f7948587ff19\") " pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:45:40.388796 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.388123    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-model-cache\") pod \"e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55\" (UID: \"96bb0499-5edf-49b8-8391-f7948587ff19\") " pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:45:40.388796 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.388367    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-home\") pod \"e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55\" (UID: \"96bb0499-5edf-49b8-8391-f7948587ff19\") " pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:45:40.641284 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.641015    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-dshm\") pod \"e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55\" (UID: \"96bb0499-5edf-49b8-8391-f7948587ff19\") " pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:45:40.688458 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.676522    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96bb0499-5edf-49b8-8391-f7948587ff19-tls-certs\") pod \"e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55\" (UID: \"96bb0499-5edf-49b8-8391-f7948587ff19\") " pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:45:40.696520 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.686426    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmk74\" (UniqueName: \"kubernetes.io/projected/96bb0499-5edf-49b8-8391-f7948587ff19-kube-api-access-jmk74\") pod \"e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55\" (UID: \"96bb0499-5edf-49b8-8391-f7948587ff19\") " pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:45:40.780587 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.780529    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx" event={"ID":"80996937-b751-47a3-a515-0bef2d5e2327","Type":"ContainerStarted","Data":"a71bf7c13f3325b6ff566b50ef7b76b4f9b28343771a154afcc7a8c80efbe2fc"}
Oct 31 01:45:40.844174 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.844115    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx" podStartSLOduration=7.844099277 podStartE2EDuration="7.844099277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:40.839229907 +0000 UTC m=+1496.842944200" watchObservedRunningTime="2026-10-31 01:45:40.844099277 +0000 UTC m=+1496.847813568"
Oct 31 01:45:40.870000 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:40.869957    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:45:41.281686 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:41.281659    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"]
Oct 31 01:45:41.549775 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:41.549682    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:41.556132 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:41.556101    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:41.790916 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:41.790875    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55" event={"ID":"96bb0499-5edf-49b8-8391-f7948587ff19","Type":"ContainerStarted","Data":"e701c9b2c22f7cf7d678bc0b0e5a19ce65c1f20c2491983596d5f82d910ac48b"}
Oct 31 01:45:41.791113 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:41.790929    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55" event={"ID":"96bb0499-5edf-49b8-8391-f7948587ff19","Type":"ContainerStarted","Data":"c7677eda7957d9a68a68b8ca491d6b16e13bc3d8e9ab75a79f5c747afa380daf"}
Oct 31 01:45:41.791184 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:41.791164    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:41.792382 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:41.792359    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:45:41.822388 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:41.822272    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55" podStartSLOduration=1.822257913 podStartE2EDuration="1.822257913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:41.821614947 +0000 UTC m=+1497.825329240" watchObservedRunningTime="2026-10-31 01:45:41.822257913 +0000 UTC m=+1497.825972204"
Oct 31 01:45:41.870515 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:41.870402    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:45:45.850698 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:45.850658    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb"]
Oct 31 01:45:45.859601 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:45.859573    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb"
Oct 31 01:45:45.869950 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:45.869909    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb"]
Oct 31 01:45:45.876182 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:45.876156    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-9cc66e-dockercfg-jswv8\""
Oct 31 01:45:45.943025 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:45.942990    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9cc66e\""
Oct 31 01:45:46.013534 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:46.012604    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a91f6a7-1cfb-483b-a5dd-6d90aafa1709-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb\" (UID: \"9a91f6a7-1cfb-483b-a5dd-6d90aafa1709\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb"
Oct 31 01:45:46.013534 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:46.012717    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8lkm\" (UniqueName: \"kubernetes.io/projected/9a91f6a7-1cfb-483b-a5dd-6d90aafa1709-kube-api-access-w8lkm\") pod \"payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb\" (UID: \"9a91f6a7-1cfb-483b-a5dd-6d90aafa1709\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb"
Oct 31 01:45:46.114139 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:46.114108    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a91f6a7-1cfb-483b-a5dd-6d90aafa1709-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb\" (UID: \"9a91f6a7-1cfb-483b-a5dd-6d90aafa1709\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb"
Oct 31 01:45:46.114504 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:46.114475    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8lkm\" (UniqueName: \"kubernetes.io/projected/9a91f6a7-1cfb-483b-a5dd-6d90aafa1709-kube-api-access-w8lkm\") pod \"payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb\" (UID: \"9a91f6a7-1cfb-483b-a5dd-6d90aafa1709\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb"
Oct 31 01:45:46.115070 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:46.115032    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a91f6a7-1cfb-483b-a5dd-6d90aafa1709-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb\" (UID: \"9a91f6a7-1cfb-483b-a5dd-6d90aafa1709\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb"
Oct 31 01:45:46.140467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:46.140038    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8lkm\" (UniqueName: \"kubernetes.io/projected/9a91f6a7-1cfb-483b-a5dd-6d90aafa1709-kube-api-access-w8lkm\") pod \"payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb\" (UID: \"9a91f6a7-1cfb-483b-a5dd-6d90aafa1709\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb"
Oct 31 01:45:46.187214 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:46.186748    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb"
Oct 31 01:45:46.498032 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:46.487621    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb"]
Oct 31 01:45:46.866525 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:46.866417    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb" event={"ID":"9a91f6a7-1cfb-483b-a5dd-6d90aafa1709","Type":"ContainerStarted","Data":"fb647be02bffc0bffe7d6e0fd8dc67317690f44e29daa7b9b6b438b141a8fb66"}
Oct 31 01:45:46.866951 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:46.866528    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb" event={"ID":"9a91f6a7-1cfb-483b-a5dd-6d90aafa1709","Type":"ContainerStarted","Data":"9ff2602b86c30a6a38d8b6e1e10a28049f5e82ccca34f79681e81a3a8e967d90"}
Oct 31 01:45:46.866951 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:46.866692    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb"
Oct 31 01:45:46.890733 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:46.890674    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb" podStartSLOduration=1.890654394 podStartE2EDuration="1.890654394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:46.887678044 +0000 UTC m=+1502.891392336" watchObservedRunningTime="2026-10-31 01:45:46.890654394 +0000 UTC m=+1502.894368686"
Oct 31 01:45:50.870711 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:50.870677    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:45:50.891830 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:50.891798    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:45:50.923293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:50.923261    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:45:54.860871 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:54.860826    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"]
Oct 31 01:45:54.864941 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:54.864915    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"
Oct 31 01:45:54.876676 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:54.876645    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vfdlj\""
Oct 31 01:45:54.876843 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:54.876696    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 01:45:54.876969 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:54.876741    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 01:45:54.876969 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:54.876781    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 01:45:54.879174 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:54.879145    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"]
Oct 31 01:45:55.024472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.023537    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3a0a419-29b7-4079-9511-96a84184790f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nbn54\" (UID: \"c3a0a419-29b7-4079-9511-96a84184790f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"
Oct 31 01:45:55.024472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.023662    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c3a0a419-29b7-4079-9511-96a84184790f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nbn54\" (UID: \"c3a0a419-29b7-4079-9511-96a84184790f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"
Oct 31 01:45:55.024472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.023713    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3a0a419-29b7-4079-9511-96a84184790f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nbn54\" (UID: \"c3a0a419-29b7-4079-9511-96a84184790f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"
Oct 31 01:45:55.024472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.023774    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3a0a419-29b7-4079-9511-96a84184790f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nbn54\" (UID: \"c3a0a419-29b7-4079-9511-96a84184790f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"
Oct 31 01:45:55.024472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.023909    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdmst\" (UniqueName: \"kubernetes.io/projected/c3a0a419-29b7-4079-9511-96a84184790f-kube-api-access-kdmst\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nbn54\" (UID: \"c3a0a419-29b7-4079-9511-96a84184790f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"
Oct 31 01:45:55.024472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.023993    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c3a0a419-29b7-4079-9511-96a84184790f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nbn54\" (UID: \"c3a0a419-29b7-4079-9511-96a84184790f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"
Oct 31 01:45:55.024472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.024029    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c3a0a419-29b7-4079-9511-96a84184790f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nbn54\" (UID: \"c3a0a419-29b7-4079-9511-96a84184790f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"
Oct 31 01:45:55.132189 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.132079    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdmst\" (UniqueName: \"kubernetes.io/projected/c3a0a419-29b7-4079-9511-96a84184790f-kube-api-access-kdmst\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nbn54\" (UID: \"c3a0a419-29b7-4079-9511-96a84184790f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"
Oct 31 01:45:55.132372 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.132205    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c3a0a419-29b7-4079-9511-96a84184790f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nbn54\" (UID: \"c3a0a419-29b7-4079-9511-96a84184790f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"
Oct 31 01:45:55.132372 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.132248    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c3a0a419-29b7-4079-9511-96a84184790f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nbn54\" (UID: \"c3a0a419-29b7-4079-9511-96a84184790f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"
Oct 31 01:45:55.132372 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.132362    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3a0a419-29b7-4079-9511-96a84184790f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nbn54\" (UID: \"c3a0a419-29b7-4079-9511-96a84184790f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"
Oct 31 01:45:55.132616 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.132511    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c3a0a419-29b7-4079-9511-96a84184790f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nbn54\" (UID: \"c3a0a419-29b7-4079-9511-96a84184790f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"
Oct 31 01:45:55.132616 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.132566    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3a0a419-29b7-4079-9511-96a84184790f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nbn54\" (UID: \"c3a0a419-29b7-4079-9511-96a84184790f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"
Oct 31 01:45:55.132722 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.132639    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3a0a419-29b7-4079-9511-96a84184790f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nbn54\" (UID: \"c3a0a419-29b7-4079-9511-96a84184790f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"
Oct 31 01:45:55.133773 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.133744    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c3a0a419-29b7-4079-9511-96a84184790f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nbn54\" (UID: \"c3a0a419-29b7-4079-9511-96a84184790f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"
Oct 31 01:45:55.137140 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.137106    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c3a0a419-29b7-4079-9511-96a84184790f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nbn54\" (UID: \"c3a0a419-29b7-4079-9511-96a84184790f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"
Oct 31 01:45:55.137296 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.137106    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3a0a419-29b7-4079-9511-96a84184790f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nbn54\" (UID: \"c3a0a419-29b7-4079-9511-96a84184790f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"
Oct 31 01:45:55.137377 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.137229    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c3a0a419-29b7-4079-9511-96a84184790f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nbn54\" (UID: \"c3a0a419-29b7-4079-9511-96a84184790f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"
Oct 31 01:45:55.138287 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.138259    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3a0a419-29b7-4079-9511-96a84184790f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nbn54\" (UID: \"c3a0a419-29b7-4079-9511-96a84184790f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"
Oct 31 01:45:55.153373 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.153338    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"]
Oct 31 01:45:55.154738 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.154702    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdmst\" (UniqueName: \"kubernetes.io/projected/c3a0a419-29b7-4079-9511-96a84184790f-kube-api-access-kdmst\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nbn54\" (UID: \"c3a0a419-29b7-4079-9511-96a84184790f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"
Oct 31 01:45:55.155909 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.155882    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3a0a419-29b7-4079-9511-96a84184790f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nbn54\" (UID: \"c3a0a419-29b7-4079-9511-96a84184790f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"
Oct 31 01:45:55.159362 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.159341    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.168351 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.168320    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"]
Oct 31 01:45:55.238755 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.238710    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"
Oct 31 01:45:55.260240 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.260195    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.260465 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.260343    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.260559 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.260501    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.260623 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.260560    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.260682 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.260635    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqcbn\" (UniqueName: \"kubernetes.io/projected/154899b1-075f-4d48-82c6-5c0fb76bff18-kube-api-access-cqcbn\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.260682 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.260671    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154899b1-075f-4d48-82c6-5c0fb76bff18-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.260787 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.260709    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.260845 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.260817    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.260901 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.260852    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.373783 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.373735    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.373783 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.373789    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.374095 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.373994    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.374170 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.374095    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.374170 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.374157    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.374285 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.374193    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.374285 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.374259    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqcbn\" (UniqueName: \"kubernetes.io/projected/154899b1-075f-4d48-82c6-5c0fb76bff18-kube-api-access-cqcbn\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.374390 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.374292    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154899b1-075f-4d48-82c6-5c0fb76bff18-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.374390 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.374327    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.376915 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.375635    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.376915 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.376132    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.377297 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.377271    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.377597 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.377573    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.378132 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.378108    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154899b1-075f-4d48-82c6-5c0fb76bff18-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.387946 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.387069    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.389325 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.389255    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.391127 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.391085    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.396677 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.396614    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqcbn\" (UniqueName: \"kubernetes.io/projected/154899b1-075f-4d48-82c6-5c0fb76bff18-kube-api-access-cqcbn\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.563931 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.563518    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:55.686268 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.685725    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"]
Oct 31 01:45:55.689512 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:45:55.689474    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a0a419_29b7_4079_9511_96a84184790f.slice/crio-b27e34e5786437dd5441a4431702e59b82cd44ad1da46907629081aa7d94502e WatchSource:0}: Error finding container b27e34e5786437dd5441a4431702e59b82cd44ad1da46907629081aa7d94502e: Status 404 returned error can't find the container with id b27e34e5786437dd5441a4431702e59b82cd44ad1da46907629081aa7d94502e
Oct 31 01:45:55.708901 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.693786    5650 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:45:55.949628 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.949304    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"]
Oct 31 01:45:55.952006 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:45:55.951978    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154899b1_075f_4d48_82c6_5c0fb76bff18.slice/crio-7f5ae611f0e4da513c7b52d379ffe15bd525382b762c3f39e9be0897cb24ec3c WatchSource:0}: Error finding container 7f5ae611f0e4da513c7b52d379ffe15bd525382b762c3f39e9be0897cb24ec3c: Status 404 returned error can't find the container with id 7f5ae611f0e4da513c7b52d379ffe15bd525382b762c3f39e9be0897cb24ec3c
Oct 31 01:45:55.955008 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.954969    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:45:55.955429 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.955403    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:45:55.955620 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.955597    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:45:55.976620 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.976589    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph" event={"ID":"154899b1-075f-4d48-82c6-5c0fb76bff18","Type":"ContainerStarted","Data":"7f5ae611f0e4da513c7b52d379ffe15bd525382b762c3f39e9be0897cb24ec3c"}
Oct 31 01:45:55.978017 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:55.977991    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54" event={"ID":"c3a0a419-29b7-4079-9511-96a84184790f","Type":"ContainerStarted","Data":"b27e34e5786437dd5441a4431702e59b82cd44ad1da46907629081aa7d94502e"}
Oct 31 01:45:56.988046 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:56.988009    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph" event={"ID":"154899b1-075f-4d48-82c6-5c0fb76bff18","Type":"ContainerStarted","Data":"c64317c691c04b77ba04327184b72a3f6d2fdd6bfcc2d80fbf3ffefd3d1880f8"}
Oct 31 01:45:57.053472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:57.040495    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph" podStartSLOduration=2.040469957 podStartE2EDuration="2.040469957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:57.01690753 +0000 UTC m=+1513.020621826" watchObservedRunningTime="2026-10-31 01:45:57.040469957 +0000 UTC m=+1513.044184250"
Oct 31 01:45:57.564403 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:57.564318    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:45:57.565707 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:57.565522    5650 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 01:45:57.565707 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:57.565579    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph" podUID="154899b1-075f-4d48-82c6-5c0fb76bff18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 01:45:57.917549 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:57.915886    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb"
Oct 31 01:45:58.564939 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:58.564896    5650 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 01:45:58.565559 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:58.565531    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph" podUID="154899b1-075f-4d48-82c6-5c0fb76bff18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 01:45:58.588266 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:45:58.588232    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:45:59.564241 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:59.564196    5650 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 01:45:59.564421 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:45:59.564264    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph" podUID="154899b1-075f-4d48-82c6-5c0fb76bff18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 01:46:00.566518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:00.565883    5650 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 01:46:00.566518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:00.565962    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph" podUID="154899b1-075f-4d48-82c6-5c0fb76bff18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 01:46:01.565320 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:01.565277    5650 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 01:46:01.565558 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:01.565347    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph" podUID="154899b1-075f-4d48-82c6-5c0fb76bff18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 01:46:02.462369 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:02.462333    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:46:02.462677 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:02.462402    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:46:02.564789 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:02.564752    5650 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 01:46:02.564933 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:02.564806    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph" podUID="154899b1-075f-4d48-82c6-5c0fb76bff18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 01:46:03.371509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:03.370580    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54" event={"ID":"c3a0a419-29b7-4079-9511-96a84184790f","Type":"ContainerStarted","Data":"98f9ac2a13aeeb29a970790d2f7bc5dfdfe4700dda5cc155a4a666015fbaa6de"}
Oct 31 01:46:03.371509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:03.371427    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"
Oct 31 01:46:03.373943 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:03.373911    5650 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-nbn54 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:46:03.374112 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:03.373968    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54" podUID="c3a0a419-29b7-4079-9511-96a84184790f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:46:03.398289 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:03.398229    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54" podStartSLOduration=2.631089342 podStartE2EDuration="9.398216583s" totalImagesPullingTime="6.767127241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:03.397833265 +0000 UTC m=+1519.401547557" watchObservedRunningTime="2026-10-31 01:46:03.398216583 +0000 UTC m=+1519.401930874"
Oct 31 01:46:03.566401 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:03.566016    5650 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 01:46:03.566401 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:03.566080    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph" podUID="154899b1-075f-4d48-82c6-5c0fb76bff18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 01:46:04.379566 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:04.379530    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nbn54"
Oct 31 01:46:04.569146 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:04.569115    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:46:04.569627 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:04.569604    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:46:04.570501 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:04.570477    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 01:46:07.410870 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:07.410833    5650 generic.go:356] "Generic (PLEG): container finished" podID="1357b02f-b722-40ac-8ec2-35f5626d8347" containerID="97cb4f814576543ee642a0a0a5995e2c4c11197cd4b9e66d10192b234e51f864" exitCode=0
Oct 31 01:46:07.411428 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:07.410913    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd" event={"ID":"1357b02f-b722-40ac-8ec2-35f5626d8347","Type":"ContainerDied","Data":"97cb4f814576543ee642a0a0a5995e2c4c11197cd4b9e66d10192b234e51f864"}
Oct 31 01:46:07.411602 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:07.411584    5650 scope.go:122] "RemoveContainer" containerID="97cb4f814576543ee642a0a0a5995e2c4c11197cd4b9e66d10192b234e51f864"
Oct 31 01:46:07.413326 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:07.413296    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:46:07.413409 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:07.413370    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:46:07.413476 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:07.413423    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:46:08.433081 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:08.433046    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd" event={"ID":"1357b02f-b722-40ac-8ec2-35f5626d8347","Type":"ContainerStarted","Data":"bb5d261e53cbb81da581b0efa07149075735abe5bb86a42ee0db05621708a1b9"}
Oct 31 01:46:08.561091 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:08.561050    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:46:08.562298 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:08.562272    5650 patch_prober.go:29] interesting pod/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 01:46:08.562411 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:08.562336    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd" podUID="1357b02f-b722-40ac-8ec2-35f5626d8347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 01:46:09.561378 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:09.561345    5650 patch_prober.go:29] interesting pod/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 01:46:09.561805 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:09.561405    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd" podUID="1357b02f-b722-40ac-8ec2-35f5626d8347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 01:46:10.561609 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:10.561564    5650 patch_prober.go:29] interesting pod/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 01:46:10.562058 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:10.561636    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd" podUID="1357b02f-b722-40ac-8ec2-35f5626d8347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 01:46:11.567466 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:11.563376    5650 patch_prober.go:29] interesting pod/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 01:46:11.567466 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:11.563440    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd" podUID="1357b02f-b722-40ac-8ec2-35f5626d8347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 01:46:12.560641 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:12.560602    5650 patch_prober.go:29] interesting pod/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 01:46:12.560946 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:12.560914    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd" podUID="1357b02f-b722-40ac-8ec2-35f5626d8347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 01:46:13.274479 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:13.272711    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:46:13.568214 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:13.565094    5650 patch_prober.go:29] interesting pod/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 01:46:13.568214 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:13.565173    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd" podUID="1357b02f-b722-40ac-8ec2-35f5626d8347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 01:46:14.758747 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:14.758330    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:46:14.760302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:14.760273    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:46:20.998711 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:20.998670    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"]
Oct 31 01:46:21.064043 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.063996    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"]
Oct 31 01:46:21.064204 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.064153    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.086232 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.086193    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"]
Oct 31 01:46:21.086406 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.086242    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"]
Oct 31 01:46:21.086406 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.086309    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.129475 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.129417    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9cc66e-openshift-default-dockercfg-9r6q4\""
Oct 31 01:46:21.227501 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.227427    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-token\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.227501 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.227501    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-podinfo\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.227745 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.227640    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ljph\" (UniqueName: \"kubernetes.io/projected/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-kube-api-access-5ljph\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.227745 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.227712    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-credential-socket\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.227827 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.227801    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-data\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.227861 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.227826    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istiod-ca-cert\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.227861 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.227852    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-podinfo\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.227939 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.227896    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-envoy\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.227979 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.227945    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-envoy\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.228058 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.228029    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-workload-certs\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.228096 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.228078    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6193b765-db07-44d7-a56a-642cafbfc2cc-istiod-ca-cert\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.228225 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.228204    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5clx\" (UniqueName: \"kubernetes.io/projected/6193b765-db07-44d7-a56a-642cafbfc2cc-kube-api-access-q5clx\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.228269 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.228254    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-credential-socket\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.228349 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.228334    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-workload-socket\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.228405 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.228389    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-token\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.228505 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.228489    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-data\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.228559 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.228532    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-workload-socket\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.228603 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.228556    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-workload-certs\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.330569 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.330430    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-workload-certs\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.330717 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.330595    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6193b765-db07-44d7-a56a-642cafbfc2cc-istiod-ca-cert\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.330717 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.330680    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5clx\" (UniqueName: \"kubernetes.io/projected/6193b765-db07-44d7-a56a-642cafbfc2cc-kube-api-access-q5clx\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.330717 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.330709    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-credential-socket\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.330846 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.330771    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-workload-socket\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.330846 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.330821    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-token\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.330972 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.330857    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-data\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.330972 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.330892    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-workload-socket\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.330972 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.330936    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-workload-certs\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.331138 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.331000    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-workload-certs\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.331138 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.331005    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-token\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.331138 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.331079    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-podinfo\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.331300 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.331140    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ljph\" (UniqueName: \"kubernetes.io/projected/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-kube-api-access-5ljph\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.331300 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.331139    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-credential-socket\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.331300 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.331158    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-workload-socket\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.331300 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.331183    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-credential-socket\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.331300 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.331277    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-data\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.331570 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.331312    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istiod-ca-cert\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.331570 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.331357    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-podinfo\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.331570 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.331389    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6193b765-db07-44d7-a56a-642cafbfc2cc-istiod-ca-cert\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.331570 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.331397    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-envoy\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.331570 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.331424    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-data\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.331570 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.331436    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-envoy\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.331570 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.331435    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-workload-socket\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.331931 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.331610    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-credential-socket\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.331931 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.331618    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-workload-certs\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.331931 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.331841    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-data\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.332431 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.332410    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istiod-ca-cert\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.333859 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.333839    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-envoy\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.334380 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.334351    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-envoy\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.334822 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.334799    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-podinfo\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.334939 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.334916    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-podinfo\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.352610 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.352535    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5clx\" (UniqueName: \"kubernetes.io/projected/6193b765-db07-44d7-a56a-642cafbfc2cc-kube-api-access-q5clx\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.362090 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.362055    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-token\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.362931 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.362830    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-token\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.365140 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.365113    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ljph\" (UniqueName: \"kubernetes.io/projected/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-kube-api-access-5ljph\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.401258 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.401222    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:21.431071 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.431028    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:21.729380 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.729346    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"]
Oct 31 01:46:21.730701 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:46:21.730673    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6193b765_db07_44d7_a56a_642cafbfc2cc.slice/crio-cc7b0cfd633e93a55d5d9dd4553e3c574eb9082e162733f06a3613a4398a7abf WatchSource:0}: Error finding container cc7b0cfd633e93a55d5d9dd4553e3c574eb9082e162733f06a3613a4398a7abf: Status 404 returned error can't find the container with id cc7b0cfd633e93a55d5d9dd4553e3c574eb9082e162733f06a3613a4398a7abf
Oct 31 01:46:21.734084 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.734051    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:46:21.734179 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.734117    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:46:21.734179 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.734144    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:46:21.823736 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:46:21.821216    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f5a1cc_1b9d_41bf_805f_99e478bfef6d.slice/crio-ed2369836bede861e4b90c2cd43158065d2a061d806cd4e99de2b665194dd9c2 WatchSource:0}: Error finding container ed2369836bede861e4b90c2cd43158065d2a061d806cd4e99de2b665194dd9c2: Status 404 returned error can't find the container with id ed2369836bede861e4b90c2cd43158065d2a061d806cd4e99de2b665194dd9c2
Oct 31 01:46:21.830970 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.830625    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:46:21.830970 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.830767    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:46:21.830970 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.830846    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:46:21.838178 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:21.836164    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"]
Oct 31 01:46:22.637471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:22.637426    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j" event={"ID":"6193b765-db07-44d7-a56a-642cafbfc2cc","Type":"ContainerStarted","Data":"d8a28229df6c0a0411fa56ffd82b4073a765668690adc6fcc9c8d82e1c0ceca9"}
Oct 31 01:46:22.656170 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:22.637498    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j" event={"ID":"6193b765-db07-44d7-a56a-642cafbfc2cc","Type":"ContainerStarted","Data":"cc7b0cfd633e93a55d5d9dd4553e3c574eb9082e162733f06a3613a4398a7abf"}
Oct 31 01:46:22.656170 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:22.639625    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx" event={"ID":"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d","Type":"ContainerStarted","Data":"78f28f62453817dfc2dba25dc258d38aabc70e53f21c2da893eca6323703fb16"}
Oct 31 01:46:22.656170 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:22.639681    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx" event={"ID":"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d","Type":"ContainerStarted","Data":"ed2369836bede861e4b90c2cd43158065d2a061d806cd4e99de2b665194dd9c2"}
Oct 31 01:46:22.694111 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:22.694036    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx" podStartSLOduration=2.694018421 podStartE2EDuration="2.694018421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:22.692803146 +0000 UTC m=+1538.696517437" watchObservedRunningTime="2026-10-31 01:46:22.694018421 +0000 UTC m=+1538.697732711"
Oct 31 01:46:22.695121 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:22.695085    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j" podStartSLOduration=2.695072596 podStartE2EDuration="2.695072596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:22.6659377 +0000 UTC m=+1538.669651990" watchObservedRunningTime="2026-10-31 01:46:22.695072596 +0000 UTC m=+1538.698786886"
Oct 31 01:46:23.420532 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:23.407867    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:23.428472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:23.426589    5650 patch_prober.go:29] interesting pod/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 01:46:23.428472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:23.426659    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j" podUID="6193b765-db07-44d7-a56a-642cafbfc2cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 01:46:23.435469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:23.432842    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:23.452217 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:23.451673    5650 patch_prober.go:29] interesting pod/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 01:46:23.452217 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:23.451747    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx" podUID="d8f5a1cc-1b9d-41bf-805f-99e478bfef6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 01:46:23.661784 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:23.661752    5650 generic.go:356] "Generic (PLEG): container finished" podID="80996937-b751-47a3-a515-0bef2d5e2327" containerID="a71bf7c13f3325b6ff566b50ef7b76b4f9b28343771a154afcc7a8c80efbe2fc" exitCode=0
Oct 31 01:46:23.662260 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:23.661798    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx" event={"ID":"80996937-b751-47a3-a515-0bef2d5e2327","Type":"ContainerDied","Data":"a71bf7c13f3325b6ff566b50ef7b76b4f9b28343771a154afcc7a8c80efbe2fc"}
Oct 31 01:46:23.662649 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:23.662632    5650 scope.go:122] "RemoveContainer" containerID="a71bf7c13f3325b6ff566b50ef7b76b4f9b28343771a154afcc7a8c80efbe2fc"
Oct 31 01:46:23.664723 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:23.664694    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:46:23.664820 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:23.664775    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:46:23.664820 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:23.664808    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:46:24.405473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:24.403912    5650 patch_prober.go:29] interesting pod/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 01:46:24.405473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:24.403982    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j" podUID="6193b765-db07-44d7-a56a-642cafbfc2cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 01:46:24.438327 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:24.438290    5650 patch_prober.go:29] interesting pod/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 01:46:24.438653 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:24.438621    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx" podUID="d8f5a1cc-1b9d-41bf-805f-99e478bfef6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 01:46:24.699405 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:24.699275    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx" event={"ID":"80996937-b751-47a3-a515-0bef2d5e2327","Type":"ContainerStarted","Data":"f015e39531a495842f2dcd6a0377feb7f7fc455cf6e87a773d8ddb7758cff2b9"}
Oct 31 01:46:25.407475 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:25.403944    5650 patch_prober.go:29] interesting pod/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 01:46:25.407475 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:25.404015    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j" podUID="6193b765-db07-44d7-a56a-642cafbfc2cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 01:46:25.436228 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:25.436077    5650 patch_prober.go:29] interesting pod/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 01:46:25.436228 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:25.436158    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx" podUID="d8f5a1cc-1b9d-41bf-805f-99e478bfef6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 01:46:25.551843 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:25.551806    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:46:25.557625 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:25.557585    5650 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:46:25.557921 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:25.557889    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx" podUID="80996937-b751-47a3-a515-0bef2d5e2327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 01:46:26.403770 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:26.403715    5650 patch_prober.go:29] interesting pod/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 01:46:26.404417 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:26.404366    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j" podUID="6193b765-db07-44d7-a56a-642cafbfc2cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 01:46:26.434948 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:26.434907    5650 patch_prober.go:29] interesting pod/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 01:46:26.435202 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:26.435175    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx" podUID="d8f5a1cc-1b9d-41bf-805f-99e478bfef6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 01:46:26.564484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:26.562903    5650 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:46:26.564484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:26.562966    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx" podUID="80996937-b751-47a3-a515-0bef2d5e2327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 01:46:26.794473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:26.793624    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:46:27.403896 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:27.403853    5650 patch_prober.go:29] interesting pod/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 01:46:27.404537 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:27.404496    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j" podUID="6193b765-db07-44d7-a56a-642cafbfc2cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 01:46:27.433836 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:27.433793    5650 patch_prober.go:29] interesting pod/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 01:46:27.434166 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:27.434123    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx" podUID="d8f5a1cc-1b9d-41bf-805f-99e478bfef6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 01:46:27.616473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:27.599422    5650 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:46:27.616473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:27.599530    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx" podUID="80996937-b751-47a3-a515-0bef2d5e2327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 01:46:28.443471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:28.420353    5650 patch_prober.go:29] interesting pod/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 01:46:28.444118 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:28.444079    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j" podUID="6193b765-db07-44d7-a56a-642cafbfc2cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 01:46:28.461508 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:28.433707    5650 patch_prober.go:29] interesting pod/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 01:46:28.462643 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:28.462582    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx" podUID="d8f5a1cc-1b9d-41bf-805f-99e478bfef6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 01:46:28.553733 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:28.553674    5650 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:46:28.553947 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:28.553760    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx" podUID="80996937-b751-47a3-a515-0bef2d5e2327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 01:46:28.589264 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:46:28.589226    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:46:29.428801 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:29.428767    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:29.430520 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:29.430058    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:29.433427 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:29.433404    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:46:29.438153 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:29.438130    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:29.438415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:29.438396    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:29.439229 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:29.439207    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:46:29.550651 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:29.550612    5650 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:46:29.551156 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:29.550673    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx" podUID="80996937-b751-47a3-a515-0bef2d5e2327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 01:46:30.554182 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:30.554151    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:46:30.555249 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:30.555225    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:46:35.114131 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:35.114093    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb"]
Oct 31 01:46:40.115606 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:40.115552    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb" podUID="9a91f6a7-1cfb-483b-a5dd-6d90aafa1709" containerName="payload-pre-processing" containerID="cri-o://fb647be02bffc0bffe7d6e0fd8dc67317690f44e29daa7b9b6b438b141a8fb66" gracePeriod=25
Oct 31 01:46:40.483428 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:40.483398    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb"
Oct 31 01:46:40.577673 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:40.577640    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a91f6a7-1cfb-483b-a5dd-6d90aafa1709-kube-api-access-w8lkm\" (UniqueName: \"kubernetes.io/projected/9a91f6a7-1cfb-483b-a5dd-6d90aafa1709-kube-api-access-w8lkm\") pod \"9a91f6a7-1cfb-483b-a5dd-6d90aafa1709\" (UID: \"9a91f6a7-1cfb-483b-a5dd-6d90aafa1709\") "
Oct 31 01:46:40.577897 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:40.577803    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a91f6a7-1cfb-483b-a5dd-6d90aafa1709-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a91f6a7-1cfb-483b-a5dd-6d90aafa1709-plugins-config-volume\") pod \"9a91f6a7-1cfb-483b-a5dd-6d90aafa1709\" (UID: \"9a91f6a7-1cfb-483b-a5dd-6d90aafa1709\") "
Oct 31 01:46:40.578285 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:40.578255    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a91f6a7-1cfb-483b-a5dd-6d90aafa1709-plugins-config-volume" pod "9a91f6a7-1cfb-483b-a5dd-6d90aafa1709" (UID: "9a91f6a7-1cfb-483b-a5dd-6d90aafa1709"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:46:40.578611 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:40.578592    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a91f6a7-1cfb-483b-a5dd-6d90aafa1709-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:46:40.581021 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:40.580996    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a91f6a7-1cfb-483b-a5dd-6d90aafa1709-kube-api-access-w8lkm" pod "9a91f6a7-1cfb-483b-a5dd-6d90aafa1709" (UID: "9a91f6a7-1cfb-483b-a5dd-6d90aafa1709"). InnerVolumeSpecName "kube-api-access-w8lkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:46:40.680056 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:40.680011    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8lkm\" (UniqueName: \"kubernetes.io/projected/9a91f6a7-1cfb-483b-a5dd-6d90aafa1709-kube-api-access-w8lkm\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:46:40.946366 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:40.946333    5650 generic.go:356] "Generic (PLEG): container finished" podID="9a91f6a7-1cfb-483b-a5dd-6d90aafa1709" containerID="fb647be02bffc0bffe7d6e0fd8dc67317690f44e29daa7b9b6b438b141a8fb66" exitCode=0
Oct 31 01:46:40.946574 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:40.946392    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb"
Oct 31 01:46:40.946574 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:40.946420    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb" event={"ID":"9a91f6a7-1cfb-483b-a5dd-6d90aafa1709","Type":"ContainerDied","Data":"fb647be02bffc0bffe7d6e0fd8dc67317690f44e29daa7b9b6b438b141a8fb66"}
Oct 31 01:46:40.946574 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:40.946486    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb" event={"ID":"9a91f6a7-1cfb-483b-a5dd-6d90aafa1709","Type":"ContainerDied","Data":"9ff2602b86c30a6a38d8b6e1e10a28049f5e82ccca34f79681e81a3a8e967d90"}
Oct 31 01:46:40.946574 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:40.946513    5650 scope.go:122] "RemoveContainer" containerID="fb647be02bffc0bffe7d6e0fd8dc67317690f44e29daa7b9b6b438b141a8fb66"
Oct 31 01:46:40.960050 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:40.960025    5650 scope.go:122] "RemoveContainer" containerID="fb647be02bffc0bffe7d6e0fd8dc67317690f44e29daa7b9b6b438b141a8fb66"
Oct 31 01:46:40.960345 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:46:40.960328    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb647be02bffc0bffe7d6e0fd8dc67317690f44e29daa7b9b6b438b141a8fb66\": container with ID starting with fb647be02bffc0bffe7d6e0fd8dc67317690f44e29daa7b9b6b438b141a8fb66 not found: ID does not exist" containerID="fb647be02bffc0bffe7d6e0fd8dc67317690f44e29daa7b9b6b438b141a8fb66"
Oct 31 01:46:40.960391 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:40.960354    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb647be02bffc0bffe7d6e0fd8dc67317690f44e29daa7b9b6b438b141a8fb66"} err="failed to get container status \"fb647be02bffc0bffe7d6e0fd8dc67317690f44e29daa7b9b6b438b141a8fb66\": rpc error: code = NotFound desc = could not find container \"fb647be02bffc0bffe7d6e0fd8dc67317690f44e29daa7b9b6b438b141a8fb66\": container with ID starting with fb647be02bffc0bffe7d6e0fd8dc67317690f44e29daa7b9b6b438b141a8fb66 not found: ID does not exist"
Oct 31 01:46:40.979675 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:40.979624    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb"]
Oct 31 01:46:40.986051 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:40.986017    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9cc66e-597bf5b795-nfjhb"]
Oct 31 01:46:42.129468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:42.129060    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a91f6a7-1cfb-483b-a5dd-6d90aafa1709" path="/var/lib/kubelet/pods/9a91f6a7-1cfb-483b-a5dd-6d90aafa1709/volumes"
Oct 31 01:46:44.432418 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:44.432371    5650 scope.go:122] "RemoveContainer" containerID="2c3f6a92285a1ae445b79ae77b6bd5d4969306f8f9a0d7bf4c012a1b595d332b"
Oct 31 01:46:44.446911 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:44.446883    5650 scope.go:122] "RemoveContainer" containerID="9bdb3c1b71e42203dcc0b74ce2383a07214b9cfd509ed97e01094999dd93dc1d"
Oct 31 01:46:58.554158 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:46:58.554114    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:46:59.914198 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:59.912049    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"]
Oct 31 01:46:59.914198 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:59.912325    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j" podUID="6193b765-db07-44d7-a56a-642cafbfc2cc" containerName="istio-proxy" containerID="cri-o://d8a28229df6c0a0411fa56ffd82b4073a765668690adc6fcc9c8d82e1c0ceca9" gracePeriod=30
Oct 31 01:46:59.916784 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:59.916755    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"]
Oct 31 01:46:59.917027 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:46:59.916983    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx" podUID="d8f5a1cc-1b9d-41bf-805f-99e478bfef6d" containerName="istio-proxy" containerID="cri-o://78f28f62453817dfc2dba25dc258d38aabc70e53f21c2da893eca6323703fb16" gracePeriod=30
Oct 31 01:47:04.026340 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.026298    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-767349a2-ph52r"]
Oct 31 01:47:04.035067 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.035033    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-767349a2-ph52r"]
Oct 31 01:47:04.119529 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.119495    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40fb5d82-5d28-4c48-acab-f5a30ea6edce" path="/var/lib/kubelet/pods/40fb5d82-5d28-4c48-acab-f5a30ea6edce/volumes"
Oct 31 01:47:04.138430 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.138397    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"]
Oct 31 01:47:04.139136 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.139119    5650 cpu_manager.go:414] "containerMap: removing container" podUID="9a91f6a7-1cfb-483b-a5dd-6d90aafa1709" containerName="payload-pre-processing"
Oct 31 01:47:04.139187 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.139137    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a91f6a7-1cfb-483b-a5dd-6d90aafa1709" containerName="payload-pre-processing"
Oct 31 01:47:04.139742 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.139229    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a91f6a7-1cfb-483b-a5dd-6d90aafa1709" containerName="payload-pre-processing"
Oct 31 01:47:04.142778 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.142759    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.154100 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.154067    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"]
Oct 31 01:47:04.235993 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.235954    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fe4d8f-d251-4852-98e4-b20e54d74df1-istiod-ca-cert\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.236195 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.236023    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-envoy\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.236195 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.236076    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-podinfo\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.236195 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.236110    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjhv2\" (UniqueName: \"kubernetes.io/projected/63fe4d8f-d251-4852-98e4-b20e54d74df1-kube-api-access-pjhv2\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.236349 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.236212    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-workload-certs\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.236349 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.236316    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-workload-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.236486 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.236353    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-data\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.236538 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.236512    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-credential-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.236595 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.236539    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-token\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.338346 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.338249    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fe4d8f-d251-4852-98e4-b20e54d74df1-istiod-ca-cert\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.338346 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.338322    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-envoy\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.338554 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.338413    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-podinfo\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.338554 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.338461    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjhv2\" (UniqueName: \"kubernetes.io/projected/63fe4d8f-d251-4852-98e4-b20e54d74df1-kube-api-access-pjhv2\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.338554 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.338512    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-workload-certs\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.338705 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.338604    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-workload-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.338705 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.338650    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-data\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.338798 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.338753    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-credential-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.338855 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.338793    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-token\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.338954 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.338928    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-workload-certs\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.339192 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.339163    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fe4d8f-d251-4852-98e4-b20e54d74df1-istiod-ca-cert\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.339277 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.339189    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-data\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.339277 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.339237    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-workload-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.339400 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.339383    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-credential-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.341219 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.341190    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-envoy\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.341485 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.341441    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-podinfo\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.374688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.361535    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-token\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.374688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.361684    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjhv2\" (UniqueName: \"kubernetes.io/projected/63fe4d8f-d251-4852-98e4-b20e54d74df1-kube-api-access-pjhv2\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.460099 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.460059    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:04.753862 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.753826    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"]
Oct 31 01:47:04.754242 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:47:04.754219    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63fe4d8f_d251_4852_98e4_b20e54d74df1.slice/crio-9dfc402ea7db51fd415f9fc3a4c1849c66280204f82e37df6f7dedd901d89d64 WatchSource:0}: Error finding container 9dfc402ea7db51fd415f9fc3a4c1849c66280204f82e37df6f7dedd901d89d64: Status 404 returned error can't find the container with id 9dfc402ea7db51fd415f9fc3a4c1849c66280204f82e37df6f7dedd901d89d64
Oct 31 01:47:04.756917 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.756881    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:47:04.757033 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.756944    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:47:04.757033 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:04.756971    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:47:05.257534 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.251385    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:47:05.260963 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.260799    5650 generic.go:356] "Generic (PLEG): container finished" podID="6193b765-db07-44d7-a56a-642cafbfc2cc" containerID="d8a28229df6c0a0411fa56ffd82b4073a765668690adc6fcc9c8d82e1c0ceca9" exitCode=0
Oct 31 01:47:05.261227 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.260976    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j" event={"ID":"6193b765-db07-44d7-a56a-642cafbfc2cc","Type":"ContainerDied","Data":"d8a28229df6c0a0411fa56ffd82b4073a765668690adc6fcc9c8d82e1c0ceca9"}
Oct 31 01:47:05.261227 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.261033    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j" event={"ID":"6193b765-db07-44d7-a56a-642cafbfc2cc","Type":"ContainerDied","Data":"cc7b0cfd633e93a55d5d9dd4553e3c574eb9082e162733f06a3613a4398a7abf"}
Oct 31 01:47:05.261227 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.261067    5650 scope.go:122] "RemoveContainer" containerID="d8a28229df6c0a0411fa56ffd82b4073a765668690adc6fcc9c8d82e1c0ceca9"
Oct 31 01:47:05.261476 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.261253    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"
Oct 31 01:47:05.266392 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.266358    5650 generic.go:356] "Generic (PLEG): container finished" podID="d8f5a1cc-1b9d-41bf-805f-99e478bfef6d" containerID="78f28f62453817dfc2dba25dc258d38aabc70e53f21c2da893eca6323703fb16" exitCode=0
Oct 31 01:47:05.266603 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.266574    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx" event={"ID":"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d","Type":"ContainerDied","Data":"78f28f62453817dfc2dba25dc258d38aabc70e53f21c2da893eca6323703fb16"}
Oct 31 01:47:05.303431 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.303390    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b" event={"ID":"63fe4d8f-d251-4852-98e4-b20e54d74df1","Type":"ContainerStarted","Data":"f81347ee6d2b080f12d62cb8146bd9f329b2cb0b61906da486351cda44669172"}
Oct 31 01:47:05.304776 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.304746    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b" event={"ID":"63fe4d8f-d251-4852-98e4-b20e54d74df1","Type":"ContainerStarted","Data":"9dfc402ea7db51fd415f9fc3a4c1849c66280204f82e37df6f7dedd901d89d64"}
Oct 31 01:47:05.312661 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.312633    5650 scope.go:122] "RemoveContainer" containerID="d8a28229df6c0a0411fa56ffd82b4073a765668690adc6fcc9c8d82e1c0ceca9"
Oct 31 01:47:05.313254 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:47:05.313230    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a28229df6c0a0411fa56ffd82b4073a765668690adc6fcc9c8d82e1c0ceca9\": container with ID starting with d8a28229df6c0a0411fa56ffd82b4073a765668690adc6fcc9c8d82e1c0ceca9 not found: ID does not exist" containerID="d8a28229df6c0a0411fa56ffd82b4073a765668690adc6fcc9c8d82e1c0ceca9"
Oct 31 01:47:05.313423 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.313398    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a28229df6c0a0411fa56ffd82b4073a765668690adc6fcc9c8d82e1c0ceca9"} err="failed to get container status \"d8a28229df6c0a0411fa56ffd82b4073a765668690adc6fcc9c8d82e1c0ceca9\": rpc error: code = NotFound desc = could not find container \"d8a28229df6c0a0411fa56ffd82b4073a765668690adc6fcc9c8d82e1c0ceca9\": container with ID starting with d8a28229df6c0a0411fa56ffd82b4073a765668690adc6fcc9c8d82e1c0ceca9 not found: ID does not exist"
Oct 31 01:47:05.356968 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.356934    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-credential-socket\") pod \"6193b765-db07-44d7-a56a-642cafbfc2cc\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") "
Oct 31 01:47:05.357277 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.357254    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-workload-socket\") pod \"6193b765-db07-44d7-a56a-642cafbfc2cc\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") "
Oct 31 01:47:05.358073 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.357397    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-credential-socket" pod "6193b765-db07-44d7-a56a-642cafbfc2cc" (UID: "6193b765-db07-44d7-a56a-642cafbfc2cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:05.358073 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.357808    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-workload-socket" pod "6193b765-db07-44d7-a56a-642cafbfc2cc" (UID: "6193b765-db07-44d7-a56a-642cafbfc2cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:05.362302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.358157    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6193b765-db07-44d7-a56a-642cafbfc2cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6193b765-db07-44d7-a56a-642cafbfc2cc-istiod-ca-cert\") pod \"6193b765-db07-44d7-a56a-642cafbfc2cc\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") "
Oct 31 01:47:05.362302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.358944    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6193b765-db07-44d7-a56a-642cafbfc2cc-istiod-ca-cert" pod "6193b765-db07-44d7-a56a-642cafbfc2cc" (UID: "6193b765-db07-44d7-a56a-642cafbfc2cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:47:05.362302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.359003    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-envoy\") pod \"6193b765-db07-44d7-a56a-642cafbfc2cc\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") "
Oct 31 01:47:05.362302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.359052    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6193b765-db07-44d7-a56a-642cafbfc2cc-kube-api-access-q5clx\" (UniqueName: \"kubernetes.io/projected/6193b765-db07-44d7-a56a-642cafbfc2cc-kube-api-access-q5clx\") pod \"6193b765-db07-44d7-a56a-642cafbfc2cc\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") "
Oct 31 01:47:05.362302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.359167    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-data\") pod \"6193b765-db07-44d7-a56a-642cafbfc2cc\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") "
Oct 31 01:47:05.362302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.359312    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-workload-certs\") pod \"6193b765-db07-44d7-a56a-642cafbfc2cc\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") "
Oct 31 01:47:05.362302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.359709    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-podinfo\") pod \"6193b765-db07-44d7-a56a-642cafbfc2cc\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") "
Oct 31 01:47:05.362302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.359792    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-token\" (UniqueName: \"kubernetes.io/projected/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-token\") pod \"6193b765-db07-44d7-a56a-642cafbfc2cc\" (UID: \"6193b765-db07-44d7-a56a-642cafbfc2cc\") "
Oct 31 01:47:05.362302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.361496    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-data" pod "6193b765-db07-44d7-a56a-642cafbfc2cc" (UID: "6193b765-db07-44d7-a56a-642cafbfc2cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:05.362302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.361513    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6193b765-db07-44d7-a56a-642cafbfc2cc-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.362302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.361552    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.362302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.361568    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.362302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.361627    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-envoy" pod "6193b765-db07-44d7-a56a-642cafbfc2cc" (UID: "6193b765-db07-44d7-a56a-642cafbfc2cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:05.362302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.362037    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-workload-certs" pod "6193b765-db07-44d7-a56a-642cafbfc2cc" (UID: "6193b765-db07-44d7-a56a-642cafbfc2cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:05.364868 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.364285    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6193b765-db07-44d7-a56a-642cafbfc2cc-kube-api-access-q5clx" pod "6193b765-db07-44d7-a56a-642cafbfc2cc" (UID: "6193b765-db07-44d7-a56a-642cafbfc2cc"). InnerVolumeSpecName "kube-api-access-q5clx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:05.365905 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.365878    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-podinfo" pod "6193b765-db07-44d7-a56a-642cafbfc2cc" (UID: "6193b765-db07-44d7-a56a-642cafbfc2cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:47:05.367810 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.367785    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-token" pod "6193b765-db07-44d7-a56a-642cafbfc2cc" (UID: "6193b765-db07-44d7-a56a-642cafbfc2cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:05.400408 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.400296    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:47:05.446260 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.446090    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b" podStartSLOduration=1.446069 podStartE2EDuration="1.446069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:05.3438504 +0000 UTC m=+1581.347564696" watchObservedRunningTime="2026-10-31 01:47:05.446069 +0000 UTC m=+1581.449783293"
Oct 31 01:47:05.462877 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.462837    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:05.463058 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.463028    5650 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 01:47:05.463141 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.463106    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b" podUID="63fe4d8f-d251-4852-98e4-b20e54d74df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 01:47:05.464293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.464259    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-credential-socket\") pod \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") "
Oct 31 01:47:05.464430 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.464347    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-envoy\") pod \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") "
Oct 31 01:47:05.464509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.464479    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-kube-api-access-5ljph\" (UniqueName: \"kubernetes.io/projected/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-kube-api-access-5ljph\") pod \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") "
Oct 31 01:47:05.464568 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.464546    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-workload-socket\") pod \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") "
Oct 31 01:47:05.464623 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.464605    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-data\") pod \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") "
Oct 31 01:47:05.464828 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.464664    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istiod-ca-cert\") pod \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") "
Oct 31 01:47:05.464828 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.464795    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-podinfo\") pod \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") "
Oct 31 01:47:05.464957 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.464886    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-workload-certs\") pod \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") "
Oct 31 01:47:05.465043 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.465025    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-token\" (UniqueName: \"kubernetes.io/projected/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-token\") pod \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\" (UID: \"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d\") "
Oct 31 01:47:05.467381 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.467349    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.467508 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.467385    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.467508 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.467409    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.467508 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.467424    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.467508 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.467441    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6193b765-db07-44d7-a56a-642cafbfc2cc-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.467508 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.467471    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5clx\" (UniqueName: \"kubernetes.io/projected/6193b765-db07-44d7-a56a-642cafbfc2cc-kube-api-access-q5clx\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.468684 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.468656    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-data" pod "d8f5a1cc-1b9d-41bf-805f-99e478bfef6d" (UID: "d8f5a1cc-1b9d-41bf-805f-99e478bfef6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:05.471590 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.471400    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istiod-ca-cert" pod "d8f5a1cc-1b9d-41bf-805f-99e478bfef6d" (UID: "d8f5a1cc-1b9d-41bf-805f-99e478bfef6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:47:05.498525 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.496572    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-credential-socket" pod "d8f5a1cc-1b9d-41bf-805f-99e478bfef6d" (UID: "d8f5a1cc-1b9d-41bf-805f-99e478bfef6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:05.498525 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.496909    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-workload-certs" pod "d8f5a1cc-1b9d-41bf-805f-99e478bfef6d" (UID: "d8f5a1cc-1b9d-41bf-805f-99e478bfef6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:05.567547 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.532584    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-workload-socket" pod "d8f5a1cc-1b9d-41bf-805f-99e478bfef6d" (UID: "d8f5a1cc-1b9d-41bf-805f-99e478bfef6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:05.722545 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.695722    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-token" pod "d8f5a1cc-1b9d-41bf-805f-99e478bfef6d" (UID: "d8f5a1cc-1b9d-41bf-805f-99e478bfef6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:05.722545 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.703170    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.722545 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.703208    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.722545 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.703230    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.722545 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.703247    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.722545 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.703267    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.722545 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.703281    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.747192 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.746631    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-envoy" pod "d8f5a1cc-1b9d-41bf-805f-99e478bfef6d" (UID: "d8f5a1cc-1b9d-41bf-805f-99e478bfef6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:05.747192 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.746742    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-podinfo" pod "d8f5a1cc-1b9d-41bf-805f-99e478bfef6d" (UID: "d8f5a1cc-1b9d-41bf-805f-99e478bfef6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:47:05.747192 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.747033    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-kube-api-access-5ljph" pod "d8f5a1cc-1b9d-41bf-805f-99e478bfef6d" (UID: "d8f5a1cc-1b9d-41bf-805f-99e478bfef6d"). InnerVolumeSpecName "kube-api-access-5ljph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:05.784147 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.784103    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"]
Oct 31 01:47:05.788722 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.788693    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-wjx8j"]
Oct 31 01:47:05.838043 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.837933    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.838267 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.838249    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ljph\" (UniqueName: \"kubernetes.io/projected/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-kube-api-access-5ljph\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:05.838348 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:05.838338    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:06.137868 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:06.137785    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6193b765-db07-44d7-a56a-642cafbfc2cc" path="/var/lib/kubelet/pods/6193b765-db07-44d7-a56a-642cafbfc2cc/volumes"
Oct 31 01:47:06.335412 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:06.335380    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"
Oct 31 01:47:06.335951 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:06.335377    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx" event={"ID":"d8f5a1cc-1b9d-41bf-805f-99e478bfef6d","Type":"ContainerDied","Data":"ed2369836bede861e4b90c2cd43158065d2a061d806cd4e99de2b665194dd9c2"}
Oct 31 01:47:06.335951 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:06.335525    5650 scope.go:122] "RemoveContainer" containerID="78f28f62453817dfc2dba25dc258d38aabc70e53f21c2da893eca6323703fb16"
Oct 31 01:47:06.365540 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:06.365505    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"]
Oct 31 01:47:06.369748 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:06.369720    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6nsxx"]
Oct 31 01:47:06.461302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:06.461269    5650 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 01:47:06.461526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:06.461327    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b" podUID="63fe4d8f-d251-4852-98e4-b20e54d74df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 01:47:07.461297 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:07.461258    5650 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 31 01:47:07.461711 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:07.461330    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b" podUID="63fe4d8f-d251-4852-98e4-b20e54d74df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 31 01:47:07.567062 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:07.567017    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb"]
Oct 31 01:47:07.588557 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:07.588519    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh"]
Oct 31 01:47:08.153272 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:08.153233    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8f5a1cc-1b9d-41bf-805f-99e478bfef6d" path="/var/lib/kubelet/pods/d8f5a1cc-1b9d-41bf-805f-99e478bfef6d/volumes"
Oct 31 01:47:08.271492 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:08.271442    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"]
Oct 31 01:47:08.272071 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:08.272038    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx" podUID="80996937-b751-47a3-a515-0bef2d5e2327" containerName="istio-proxy" containerID="cri-o://f015e39531a495842f2dcd6a0377feb7f7fc455cf6e87a773d8ddb7758cff2b9" gracePeriod=30
Oct 31 01:47:08.276812 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:08.276781    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"]
Oct 31 01:47:08.366411 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:08.361993    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b" podUID="63fe4d8f-d251-4852-98e4-b20e54d74df1" containerName="istio-proxy" containerID="cri-o://f81347ee6d2b080f12d62cb8146bd9f329b2cb0b61906da486351cda44669172" gracePeriod=30
Oct 31 01:47:09.261577 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:09.261540    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-271413cd-q9q2c"]
Oct 31 01:47:09.262210 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:09.262193    5650 cpu_manager.go:414] "containerMap: removing container" podUID="6193b765-db07-44d7-a56a-642cafbfc2cc" containerName="istio-proxy"
Oct 31 01:47:09.262210 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:09.262210    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6193b765-db07-44d7-a56a-642cafbfc2cc" containerName="istio-proxy"
Oct 31 01:47:09.262299 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:09.262252    5650 cpu_manager.go:414] "containerMap: removing container" podUID="d8f5a1cc-1b9d-41bf-805f-99e478bfef6d" containerName="istio-proxy"
Oct 31 01:47:09.262299 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:09.262257    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8f5a1cc-1b9d-41bf-805f-99e478bfef6d" containerName="istio-proxy"
Oct 31 01:47:09.262379 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:09.262368    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="6193b765-db07-44d7-a56a-642cafbfc2cc" containerName="istio-proxy"
Oct 31 01:47:09.262414 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:09.262388    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8f5a1cc-1b9d-41bf-805f-99e478bfef6d" containerName="istio-proxy"
Oct 31 01:47:09.267294 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:09.267273    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271413cd-q9q2c"
Oct 31 01:47:09.272630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:09.272602    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:47:09.272781 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:09.272717    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:47:09.283233 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:09.283107    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-271413cd-q9q2c"]
Oct 31 01:47:09.355608 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:09.355569    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq5jr\" (UniqueName: \"kubernetes.io/projected/3d94739d-327b-4ece-9b0a-bf38caeae06e-kube-api-access-dq5jr\") pod \"maas-api-revoke-sub-271413cd-q9q2c\" (UID: \"3d94739d-327b-4ece-9b0a-bf38caeae06e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271413cd-q9q2c"
Oct 31 01:47:09.355782 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:09.355699    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d94739d-327b-4ece-9b0a-bf38caeae06e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-271413cd-q9q2c\" (UID: \"3d94739d-327b-4ece-9b0a-bf38caeae06e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271413cd-q9q2c"
Oct 31 01:47:09.355950 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:09.355914    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d94739d-327b-4ece-9b0a-bf38caeae06e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-271413cd-q9q2c\" (UID: \"3d94739d-327b-4ece-9b0a-bf38caeae06e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271413cd-q9q2c"
Oct 31 01:47:09.457379 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:09.457334    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d94739d-327b-4ece-9b0a-bf38caeae06e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-271413cd-q9q2c\" (UID: \"3d94739d-327b-4ece-9b0a-bf38caeae06e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271413cd-q9q2c"
Oct 31 01:47:09.457587 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:09.457435    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq5jr\" (UniqueName: \"kubernetes.io/projected/3d94739d-327b-4ece-9b0a-bf38caeae06e-kube-api-access-dq5jr\") pod \"maas-api-revoke-sub-271413cd-q9q2c\" (UID: \"3d94739d-327b-4ece-9b0a-bf38caeae06e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271413cd-q9q2c"
Oct 31 01:47:09.457587 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:09.457508    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d94739d-327b-4ece-9b0a-bf38caeae06e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-271413cd-q9q2c\" (UID: \"3d94739d-327b-4ece-9b0a-bf38caeae06e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271413cd-q9q2c"
Oct 31 01:47:09.458110 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:09.458087    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d94739d-327b-4ece-9b0a-bf38caeae06e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-271413cd-q9q2c\" (UID: \"3d94739d-327b-4ece-9b0a-bf38caeae06e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271413cd-q9q2c"
Oct 31 01:47:09.459916 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:09.459894    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d94739d-327b-4ece-9b0a-bf38caeae06e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-271413cd-q9q2c\" (UID: \"3d94739d-327b-4ece-9b0a-bf38caeae06e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271413cd-q9q2c"
Oct 31 01:47:09.470487 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:09.470433    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq5jr\" (UniqueName: \"kubernetes.io/projected/3d94739d-327b-4ece-9b0a-bf38caeae06e-kube-api-access-dq5jr\") pod \"maas-api-revoke-sub-271413cd-q9q2c\" (UID: \"3d94739d-327b-4ece-9b0a-bf38caeae06e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271413cd-q9q2c"
Oct 31 01:47:09.587692 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:09.587600    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271413cd-q9q2c"
Oct 31 01:47:09.950871 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:09.950837    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-271413cd-q9q2c"]
Oct 31 01:47:09.951021 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:47:09.950960    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d94739d_327b_4ece_9b0a_bf38caeae06e.slice/crio-b76ca328ce197f678d7d7ff1dc637dfb23a5728f6a5fc5f35751bd6b34a16359 WatchSource:0}: Error finding container b76ca328ce197f678d7d7ff1dc637dfb23a5728f6a5fc5f35751bd6b34a16359: Status 404 returned error can't find the container with id b76ca328ce197f678d7d7ff1dc637dfb23a5728f6a5fc5f35751bd6b34a16359
Oct 31 01:47:10.386679 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:10.386645    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271413cd-q9q2c" event={"ID":"3d94739d-327b-4ece-9b0a-bf38caeae06e","Type":"ContainerStarted","Data":"b76ca328ce197f678d7d7ff1dc637dfb23a5728f6a5fc5f35751bd6b34a16359"}
Oct 31 01:47:11.396985 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:11.396887    5650 generic.go:356] "Generic (PLEG): container finished" podID="3d94739d-327b-4ece-9b0a-bf38caeae06e" containerID="a72b91c2a65bf263eae4002e7d1496fda1e1932a04af55ecdf1868289e379346" exitCode=0
Oct 31 01:47:11.397510 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:11.396988    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271413cd-q9q2c" event={"ID":"3d94739d-327b-4ece-9b0a-bf38caeae06e","Type":"ContainerDied","Data":"a72b91c2a65bf263eae4002e7d1496fda1e1932a04af55ecdf1868289e379346"}
Oct 31 01:47:12.567620 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:12.567589    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb" podUID="2946cec0-bd91-40ac-a095-6e174a3bf3b2" containerName="payload-processing" containerID="cri-o://4ae41fe256e8d20d8b3da54e28e028ee2cc4a10f3db18521a6d35c26094abbbb" gracePeriod=25
Oct 31 01:47:12.589129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:12.589083    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh" podUID="d359baf9-7e49-4e89-afb1-7c763722c087" containerName="payload-pre-processing" containerID="cri-o://24137e3dc6c858268fb4332653f93731f7f851fd1bf525ed59f94359ddbc5c50" gracePeriod=25
Oct 31 01:47:12.601897 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:12.601871    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271413cd-q9q2c"
Oct 31 01:47:12.713549 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:12.713512    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d94739d-327b-4ece-9b0a-bf38caeae06e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d94739d-327b-4ece-9b0a-bf38caeae06e-maas-api-xks-ca\") pod \"3d94739d-327b-4ece-9b0a-bf38caeae06e\" (UID: \"3d94739d-327b-4ece-9b0a-bf38caeae06e\") "
Oct 31 01:47:12.713730 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:12.713585    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d94739d-327b-4ece-9b0a-bf38caeae06e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d94739d-327b-4ece-9b0a-bf38caeae06e-maas-api-openshift-ca\") pod \"3d94739d-327b-4ece-9b0a-bf38caeae06e\" (UID: \"3d94739d-327b-4ece-9b0a-bf38caeae06e\") "
Oct 31 01:47:12.713819 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:12.713806    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d94739d-327b-4ece-9b0a-bf38caeae06e-kube-api-access-dq5jr\" (UniqueName: \"kubernetes.io/projected/3d94739d-327b-4ece-9b0a-bf38caeae06e-kube-api-access-dq5jr\") pod \"3d94739d-327b-4ece-9b0a-bf38caeae06e\" (UID: \"3d94739d-327b-4ece-9b0a-bf38caeae06e\") "
Oct 31 01:47:12.713942 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:12.713918    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d94739d-327b-4ece-9b0a-bf38caeae06e-maas-api-openshift-ca" pod "3d94739d-327b-4ece-9b0a-bf38caeae06e" (UID: "3d94739d-327b-4ece-9b0a-bf38caeae06e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:47:12.714247 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:12.714231    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d94739d-327b-4ece-9b0a-bf38caeae06e-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:12.715681 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:12.715661    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d94739d-327b-4ece-9b0a-bf38caeae06e-maas-api-xks-ca" pod "3d94739d-327b-4ece-9b0a-bf38caeae06e" (UID: "3d94739d-327b-4ece-9b0a-bf38caeae06e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:47:12.715956 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:12.715933    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d94739d-327b-4ece-9b0a-bf38caeae06e-kube-api-access-dq5jr" pod "3d94739d-327b-4ece-9b0a-bf38caeae06e" (UID: "3d94739d-327b-4ece-9b0a-bf38caeae06e"). InnerVolumeSpecName "kube-api-access-dq5jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:12.815656 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:12.815617    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq5jr\" (UniqueName: \"kubernetes.io/projected/3d94739d-327b-4ece-9b0a-bf38caeae06e-kube-api-access-dq5jr\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:12.815656 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:12.815656    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d94739d-327b-4ece-9b0a-bf38caeae06e-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.433724 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.433686    5650 generic.go:356] "Generic (PLEG): container finished" podID="80996937-b751-47a3-a515-0bef2d5e2327" containerID="f015e39531a495842f2dcd6a0377feb7f7fc455cf6e87a773d8ddb7758cff2b9" exitCode=0
Oct 31 01:47:13.433921 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.433765    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx" event={"ID":"80996937-b751-47a3-a515-0bef2d5e2327","Type":"ContainerDied","Data":"f015e39531a495842f2dcd6a0377feb7f7fc455cf6e87a773d8ddb7758cff2b9"}
Oct 31 01:47:13.433921 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.433861    5650 scope.go:122] "RemoveContainer" containerID="a71bf7c13f3325b6ff566b50ef7b76b4f9b28343771a154afcc7a8c80efbe2fc"
Oct 31 01:47:13.435623 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.435598    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271413cd-q9q2c" event={"ID":"3d94739d-327b-4ece-9b0a-bf38caeae06e","Type":"ContainerDied","Data":"b76ca328ce197f678d7d7ff1dc637dfb23a5728f6a5fc5f35751bd6b34a16359"}
Oct 31 01:47:13.435751 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.435639    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b76ca328ce197f678d7d7ff1dc637dfb23a5728f6a5fc5f35751bd6b34a16359"
Oct 31 01:47:13.435751 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.435638    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271413cd-q9q2c"
Oct 31 01:47:13.546788 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.546717    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:47:13.730879 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.730842    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80996937-b751-47a3-a515-0bef2d5e2327-kube-api-access-wlbqc\" (UniqueName: \"kubernetes.io/projected/80996937-b751-47a3-a515-0bef2d5e2327-kube-api-access-wlbqc\") pod \"80996937-b751-47a3-a515-0bef2d5e2327\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") "
Oct 31 01:47:13.731323 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.730971    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-istio-envoy\") pod \"80996937-b751-47a3-a515-0bef2d5e2327\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") "
Oct 31 01:47:13.731323 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.731024    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-credential-socket\") pod \"80996937-b751-47a3-a515-0bef2d5e2327\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") "
Oct 31 01:47:13.731323 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.731078    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80996937-b751-47a3-a515-0bef2d5e2327-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80996937-b751-47a3-a515-0bef2d5e2327-istio-podinfo\") pod \"80996937-b751-47a3-a515-0bef2d5e2327\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") "
Oct 31 01:47:13.731323 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.731150    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80996937-b751-47a3-a515-0bef2d5e2327-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80996937-b751-47a3-a515-0bef2d5e2327-istiod-ca-cert\") pod \"80996937-b751-47a3-a515-0bef2d5e2327\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") "
Oct 31 01:47:13.731323 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.731190    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-workload-socket\") pod \"80996937-b751-47a3-a515-0bef2d5e2327\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") "
Oct 31 01:47:13.731323 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.731255    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-istio-data\") pod \"80996937-b751-47a3-a515-0bef2d5e2327\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") "
Oct 31 01:47:13.731323 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.731306    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80996937-b751-47a3-a515-0bef2d5e2327-istio-token\" (UniqueName: \"kubernetes.io/projected/80996937-b751-47a3-a515-0bef2d5e2327-istio-token\") pod \"80996937-b751-47a3-a515-0bef2d5e2327\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") "
Oct 31 01:47:13.731726 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.731380    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-credential-socket" pod "80996937-b751-47a3-a515-0bef2d5e2327" (UID: "80996937-b751-47a3-a515-0bef2d5e2327"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:13.731726 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.731503    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-workload-certs\") pod \"80996937-b751-47a3-a515-0bef2d5e2327\" (UID: \"80996937-b751-47a3-a515-0bef2d5e2327\") "
Oct 31 01:47:13.731726 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.731615    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-istio-data" pod "80996937-b751-47a3-a515-0bef2d5e2327" (UID: "80996937-b751-47a3-a515-0bef2d5e2327"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:13.731726 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.731631    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-workload-socket" pod "80996937-b751-47a3-a515-0bef2d5e2327" (UID: "80996937-b751-47a3-a515-0bef2d5e2327"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:13.731906 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.731882    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-workload-certs" pod "80996937-b751-47a3-a515-0bef2d5e2327" (UID: "80996937-b751-47a3-a515-0bef2d5e2327"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:13.732100 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.732077    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80996937-b751-47a3-a515-0bef2d5e2327-istiod-ca-cert" pod "80996937-b751-47a3-a515-0bef2d5e2327" (UID: "80996937-b751-47a3-a515-0bef2d5e2327"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:47:13.732310 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.732290    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.732383 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.732316    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.732383 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.732330    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80996937-b751-47a3-a515-0bef2d5e2327-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.732383 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.732344    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.732383 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.732358    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.733253 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.733226    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-istio-envoy" pod "80996937-b751-47a3-a515-0bef2d5e2327" (UID: "80996937-b751-47a3-a515-0bef2d5e2327"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:13.733485 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.733423    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80996937-b751-47a3-a515-0bef2d5e2327-istio-podinfo" pod "80996937-b751-47a3-a515-0bef2d5e2327" (UID: "80996937-b751-47a3-a515-0bef2d5e2327"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:47:13.733727 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.733701    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80996937-b751-47a3-a515-0bef2d5e2327-kube-api-access-wlbqc" pod "80996937-b751-47a3-a515-0bef2d5e2327" (UID: "80996937-b751-47a3-a515-0bef2d5e2327"). InnerVolumeSpecName "kube-api-access-wlbqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:13.733815 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.733746    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80996937-b751-47a3-a515-0bef2d5e2327-istio-token" pod "80996937-b751-47a3-a515-0bef2d5e2327" (UID: "80996937-b751-47a3-a515-0bef2d5e2327"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:13.836223 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.834922    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80996937-b751-47a3-a515-0bef2d5e2327-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.836223 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.834962    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlbqc\" (UniqueName: \"kubernetes.io/projected/80996937-b751-47a3-a515-0bef2d5e2327-kube-api-access-wlbqc\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.836223 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.835005    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80996937-b751-47a3-a515-0bef2d5e2327-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:13.836223 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:13.835020    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80996937-b751-47a3-a515-0bef2d5e2327-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:14.445173 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:14.445078    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"
Oct 31 01:47:14.445173 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:14.445119    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx" event={"ID":"80996937-b751-47a3-a515-0bef2d5e2327","Type":"ContainerDied","Data":"41b39ca463e59e2824893463401265b428c4a594ab95f9540fc4ff24936e59f0"}
Oct 31 01:47:14.445403 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:14.445179    5650 scope.go:122] "RemoveContainer" containerID="f015e39531a495842f2dcd6a0377feb7f7fc455cf6e87a773d8ddb7758cff2b9"
Oct 31 01:47:14.471329 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:14.471296    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"]
Oct 31 01:47:14.478124 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:14.478090    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-dw9wx"]
Oct 31 01:47:16.119276 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.119239    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80996937-b751-47a3-a515-0bef2d5e2327" path="/var/lib/kubelet/pods/80996937-b751-47a3-a515-0bef2d5e2327/volumes"
Oct 31 01:47:16.481703 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.481669    5650 generic.go:356] "Generic (PLEG): container finished" podID="63fe4d8f-d251-4852-98e4-b20e54d74df1" containerID="f81347ee6d2b080f12d62cb8146bd9f329b2cb0b61906da486351cda44669172" exitCode=0
Oct 31 01:47:16.481869 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.481736    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b" event={"ID":"63fe4d8f-d251-4852-98e4-b20e54d74df1","Type":"ContainerDied","Data":"f81347ee6d2b080f12d62cb8146bd9f329b2cb0b61906da486351cda44669172"}
Oct 31 01:47:16.507867 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.507841    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:16.582315 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.582264    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-workload-socket\") pod \"63fe4d8f-d251-4852-98e4-b20e54d74df1\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") "
Oct 31 01:47:16.582523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.582329    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-workload-certs\") pod \"63fe4d8f-d251-4852-98e4-b20e54d74df1\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") "
Oct 31 01:47:16.582523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.582407    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63fe4d8f-d251-4852-98e4-b20e54d74df1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fe4d8f-d251-4852-98e4-b20e54d74df1-istiod-ca-cert\") pod \"63fe4d8f-d251-4852-98e4-b20e54d74df1\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") "
Oct 31 01:47:16.582523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.582485    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-podinfo\") pod \"63fe4d8f-d251-4852-98e4-b20e54d74df1\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") "
Oct 31 01:47:16.582523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.582520    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-credential-socket\") pod \"63fe4d8f-d251-4852-98e4-b20e54d74df1\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") "
Oct 31 01:47:16.582754 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.582583    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-data\") pod \"63fe4d8f-d251-4852-98e4-b20e54d74df1\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") "
Oct 31 01:47:16.582754 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.582604    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-workload-socket" pod "63fe4d8f-d251-4852-98e4-b20e54d74df1" (UID: "63fe4d8f-d251-4852-98e4-b20e54d74df1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:16.582754 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.582638    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-token\" (UniqueName: \"kubernetes.io/projected/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-token\") pod \"63fe4d8f-d251-4852-98e4-b20e54d74df1\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") "
Oct 31 01:47:16.582754 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.582692    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63fe4d8f-d251-4852-98e4-b20e54d74df1-kube-api-access-pjhv2\" (UniqueName: \"kubernetes.io/projected/63fe4d8f-d251-4852-98e4-b20e54d74df1-kube-api-access-pjhv2\") pod \"63fe4d8f-d251-4852-98e4-b20e54d74df1\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") "
Oct 31 01:47:16.582960 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.582756    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-workload-certs" pod "63fe4d8f-d251-4852-98e4-b20e54d74df1" (UID: "63fe4d8f-d251-4852-98e4-b20e54d74df1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:16.582960 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.582800    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-envoy\") pod \"63fe4d8f-d251-4852-98e4-b20e54d74df1\" (UID: \"63fe4d8f-d251-4852-98e4-b20e54d74df1\") "
Oct 31 01:47:16.582960 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.582813    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-credential-socket" pod "63fe4d8f-d251-4852-98e4-b20e54d74df1" (UID: "63fe4d8f-d251-4852-98e4-b20e54d74df1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:16.583110 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.582964    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-data" pod "63fe4d8f-d251-4852-98e4-b20e54d74df1" (UID: "63fe4d8f-d251-4852-98e4-b20e54d74df1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:16.583524 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.583498    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:16.583634 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.583528    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:16.583634 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.583532    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63fe4d8f-d251-4852-98e4-b20e54d74df1-istiod-ca-cert" pod "63fe4d8f-d251-4852-98e4-b20e54d74df1" (UID: "63fe4d8f-d251-4852-98e4-b20e54d74df1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:47:16.583634 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.583544    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:16.583634 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.583558    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:16.586377 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.586345    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-envoy" pod "63fe4d8f-d251-4852-98e4-b20e54d74df1" (UID: "63fe4d8f-d251-4852-98e4-b20e54d74df1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:47:16.586525 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.586462    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-podinfo" pod "63fe4d8f-d251-4852-98e4-b20e54d74df1" (UID: "63fe4d8f-d251-4852-98e4-b20e54d74df1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:47:16.586525 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.586502    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-token" pod "63fe4d8f-d251-4852-98e4-b20e54d74df1" (UID: "63fe4d8f-d251-4852-98e4-b20e54d74df1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:16.587600 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.587576    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63fe4d8f-d251-4852-98e4-b20e54d74df1-kube-api-access-pjhv2" pod "63fe4d8f-d251-4852-98e4-b20e54d74df1" (UID: "63fe4d8f-d251-4852-98e4-b20e54d74df1"). InnerVolumeSpecName "kube-api-access-pjhv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:16.685148 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.685097    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:16.685148 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.685140    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:16.685148 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.685152    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjhv2\" (UniqueName: \"kubernetes.io/projected/63fe4d8f-d251-4852-98e4-b20e54d74df1-kube-api-access-pjhv2\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:16.685148 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.685162    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63fe4d8f-d251-4852-98e4-b20e54d74df1-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:16.685427 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:16.685171    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63fe4d8f-d251-4852-98e4-b20e54d74df1-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:17.297257 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:17.297211    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb" podUID="2946cec0-bd91-40ac-a095-6e174a3bf3b2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:47:17.297734 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:17.297211    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh" podUID="d359baf9-7e49-4e89-afb1-7c763722c087" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:47:17.493888 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:17.493857    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"
Oct 31 01:47:17.494031 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:17.493856    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b" event={"ID":"63fe4d8f-d251-4852-98e4-b20e54d74df1","Type":"ContainerDied","Data":"9dfc402ea7db51fd415f9fc3a4c1849c66280204f82e37df6f7dedd901d89d64"}
Oct 31 01:47:17.494031 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:17.494009    5650 scope.go:122] "RemoveContainer" containerID="f81347ee6d2b080f12d62cb8146bd9f329b2cb0b61906da486351cda44669172"
Oct 31 01:47:17.529502 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:17.529185    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"]
Oct 31 01:47:17.536121 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:17.536087    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-rmf5b"]
Oct 31 01:47:18.119738 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:18.119707    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63fe4d8f-d251-4852-98e4-b20e54d74df1" path="/var/lib/kubelet/pods/63fe4d8f-d251-4852-98e4-b20e54d74df1/volumes"
Oct 31 01:47:25.963500 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:25.950952    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55" podUID="96bb0499-5edf-49b8-8391-f7948587ff19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:25.963500 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:25.955206    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:25.987467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:25.981836    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:25.987467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:25.984692    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 01:47:26.072618 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:26.057329    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 01:47:26.711387 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:26.711342    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 31 01:47:26.715289 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:26.711616    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 01:47:27.297839 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:27.297801    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb" podUID="2946cec0-bd91-40ac-a095-6e174a3bf3b2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:47:27.297839 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:27.297820    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh" podUID="d359baf9-7e49-4e89-afb1-7c763722c087" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:47:28.554405 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:47:28.554353    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:47:32.945922 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:32.945888    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb"
Oct 31 01:47:33.017972 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.017937    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh"
Oct 31 01:47:33.029440 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.029411    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2946cec0-bd91-40ac-a095-6e174a3bf3b2-kube-api-access-vdt4k\" (UniqueName: \"kubernetes.io/projected/2946cec0-bd91-40ac-a095-6e174a3bf3b2-kube-api-access-vdt4k\") pod \"2946cec0-bd91-40ac-a095-6e174a3bf3b2\" (UID: \"2946cec0-bd91-40ac-a095-6e174a3bf3b2\") "
Oct 31 01:47:33.029607 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.029481    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2946cec0-bd91-40ac-a095-6e174a3bf3b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2946cec0-bd91-40ac-a095-6e174a3bf3b2-plugins-config-volume\") pod \"2946cec0-bd91-40ac-a095-6e174a3bf3b2\" (UID: \"2946cec0-bd91-40ac-a095-6e174a3bf3b2\") "
Oct 31 01:47:33.029893 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.029869    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2946cec0-bd91-40ac-a095-6e174a3bf3b2-plugins-config-volume" pod "2946cec0-bd91-40ac-a095-6e174a3bf3b2" (UID: "2946cec0-bd91-40ac-a095-6e174a3bf3b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:47:33.033711 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.030567    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2946cec0-bd91-40ac-a095-6e174a3bf3b2-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:33.043608 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.043387    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2946cec0-bd91-40ac-a095-6e174a3bf3b2-kube-api-access-vdt4k" pod "2946cec0-bd91-40ac-a095-6e174a3bf3b2" (UID: "2946cec0-bd91-40ac-a095-6e174a3bf3b2"). InnerVolumeSpecName "kube-api-access-vdt4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:33.131715 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.131617    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d359baf9-7e49-4e89-afb1-7c763722c087-kube-api-access-7p7kd\" (UniqueName: \"kubernetes.io/projected/d359baf9-7e49-4e89-afb1-7c763722c087-kube-api-access-7p7kd\") pod \"d359baf9-7e49-4e89-afb1-7c763722c087\" (UID: \"d359baf9-7e49-4e89-afb1-7c763722c087\") "
Oct 31 01:47:33.131891 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.131753    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d359baf9-7e49-4e89-afb1-7c763722c087-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d359baf9-7e49-4e89-afb1-7c763722c087-plugins-config-volume\") pod \"d359baf9-7e49-4e89-afb1-7c763722c087\" (UID: \"d359baf9-7e49-4e89-afb1-7c763722c087\") "
Oct 31 01:47:33.132128 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.132103    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d359baf9-7e49-4e89-afb1-7c763722c087-plugins-config-volume" pod "d359baf9-7e49-4e89-afb1-7c763722c087" (UID: "d359baf9-7e49-4e89-afb1-7c763722c087"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:47:33.132197 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.132137    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdt4k\" (UniqueName: \"kubernetes.io/projected/2946cec0-bd91-40ac-a095-6e174a3bf3b2-kube-api-access-vdt4k\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:33.133841 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.133818    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d359baf9-7e49-4e89-afb1-7c763722c087-kube-api-access-7p7kd" pod "d359baf9-7e49-4e89-afb1-7c763722c087" (UID: "d359baf9-7e49-4e89-afb1-7c763722c087"). InnerVolumeSpecName "kube-api-access-7p7kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:33.243547 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.243517    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p7kd\" (UniqueName: \"kubernetes.io/projected/d359baf9-7e49-4e89-afb1-7c763722c087-kube-api-access-7p7kd\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:33.243547 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.243544    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d359baf9-7e49-4e89-afb1-7c763722c087-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:47:33.464492 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.457890    5650 generic.go:356] "Generic (PLEG): container finished" podID="d359baf9-7e49-4e89-afb1-7c763722c087" containerID="24137e3dc6c858268fb4332653f93731f7f851fd1bf525ed59f94359ddbc5c50" exitCode=0
Oct 31 01:47:33.464492 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.458221    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh"
Oct 31 01:47:33.465518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.465473    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh" event={"ID":"d359baf9-7e49-4e89-afb1-7c763722c087","Type":"ContainerDied","Data":"24137e3dc6c858268fb4332653f93731f7f851fd1bf525ed59f94359ddbc5c50"}
Oct 31 01:47:33.465688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.465558    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh" event={"ID":"d359baf9-7e49-4e89-afb1-7c763722c087","Type":"ContainerDied","Data":"37cf62a4dbba4e748c18236aa03edf611a4d35a7eb3de23f40d2e089b088fe52"}
Oct 31 01:47:33.465688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.465590    5650 scope.go:122] "RemoveContainer" containerID="24137e3dc6c858268fb4332653f93731f7f851fd1bf525ed59f94359ddbc5c50"
Oct 31 01:47:33.475538 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.475504    5650 generic.go:356] "Generic (PLEG): container finished" podID="2946cec0-bd91-40ac-a095-6e174a3bf3b2" containerID="4ae41fe256e8d20d8b3da54e28e028ee2cc4a10f3db18521a6d35c26094abbbb" exitCode=0
Oct 31 01:47:33.475684 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.475571    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb" event={"ID":"2946cec0-bd91-40ac-a095-6e174a3bf3b2","Type":"ContainerDied","Data":"4ae41fe256e8d20d8b3da54e28e028ee2cc4a10f3db18521a6d35c26094abbbb"}
Oct 31 01:47:33.475684 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.475678    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb"
Oct 31 01:47:33.476564 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.476532    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb" event={"ID":"2946cec0-bd91-40ac-a095-6e174a3bf3b2","Type":"ContainerDied","Data":"20cf91b1823e235eab7e6ef19b36b076bce7fad09b9e4e29b316ba82c8ad5fd6"}
Oct 31 01:47:33.488299 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.488277    5650 scope.go:122] "RemoveContainer" containerID="24137e3dc6c858268fb4332653f93731f7f851fd1bf525ed59f94359ddbc5c50"
Oct 31 01:47:33.488765 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:47:33.488694    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24137e3dc6c858268fb4332653f93731f7f851fd1bf525ed59f94359ddbc5c50\": container with ID starting with 24137e3dc6c858268fb4332653f93731f7f851fd1bf525ed59f94359ddbc5c50 not found: ID does not exist" containerID="24137e3dc6c858268fb4332653f93731f7f851fd1bf525ed59f94359ddbc5c50"
Oct 31 01:47:33.488765 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.488741    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24137e3dc6c858268fb4332653f93731f7f851fd1bf525ed59f94359ddbc5c50"} err="failed to get container status \"24137e3dc6c858268fb4332653f93731f7f851fd1bf525ed59f94359ddbc5c50\": rpc error: code = NotFound desc = could not find container \"24137e3dc6c858268fb4332653f93731f7f851fd1bf525ed59f94359ddbc5c50\": container with ID starting with 24137e3dc6c858268fb4332653f93731f7f851fd1bf525ed59f94359ddbc5c50 not found: ID does not exist"
Oct 31 01:47:33.488765 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.488766    5650 scope.go:122] "RemoveContainer" containerID="4ae41fe256e8d20d8b3da54e28e028ee2cc4a10f3db18521a6d35c26094abbbb"
Oct 31 01:47:33.507312 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.507277    5650 scope.go:122] "RemoveContainer" containerID="4ae41fe256e8d20d8b3da54e28e028ee2cc4a10f3db18521a6d35c26094abbbb"
Oct 31 01:47:33.507909 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:47:33.507857    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ae41fe256e8d20d8b3da54e28e028ee2cc4a10f3db18521a6d35c26094abbbb\": container with ID starting with 4ae41fe256e8d20d8b3da54e28e028ee2cc4a10f3db18521a6d35c26094abbbb not found: ID does not exist" containerID="4ae41fe256e8d20d8b3da54e28e028ee2cc4a10f3db18521a6d35c26094abbbb"
Oct 31 01:47:33.507909 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.507891    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh"]
Oct 31 01:47:33.507909 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.507897    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae41fe256e8d20d8b3da54e28e028ee2cc4a10f3db18521a6d35c26094abbbb"} err="failed to get container status \"4ae41fe256e8d20d8b3da54e28e028ee2cc4a10f3db18521a6d35c26094abbbb\": rpc error: code = NotFound desc = could not find container \"4ae41fe256e8d20d8b3da54e28e028ee2cc4a10f3db18521a6d35c26094abbbb\": container with ID starting with 4ae41fe256e8d20d8b3da54e28e028ee2cc4a10f3db18521a6d35c26094abbbb not found: ID does not exist"
Oct 31 01:47:33.514099 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.514061    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb5756cb-7885d746b7-jxnxh"]
Oct 31 01:47:33.525539 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.525502    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb"]
Oct 31 01:47:33.529549 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:33.529522    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb5756cb-86dd696cc-jkhdb"]
Oct 31 01:47:34.120967 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:34.120936    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2946cec0-bd91-40ac-a095-6e174a3bf3b2" path="/var/lib/kubelet/pods/2946cec0-bd91-40ac-a095-6e174a3bf3b2/volumes"
Oct 31 01:47:34.121425 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:34.121410    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d359baf9-7e49-4e89-afb1-7c763722c087" path="/var/lib/kubelet/pods/d359baf9-7e49-4e89-afb1-7c763722c087/volumes"
Oct 31 01:47:52.610683 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:52.610631    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:52.695517 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:52.695474    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:52.892704 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:52.892590    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:52.894643 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:52.894604    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:52.913475 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:52.912556    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 01:47:52.926509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:52.925667    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 01:47:52.927348 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:52.927307    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 01:47:53.193524 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:53.188963    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:47:56.332272 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.332170    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"]
Oct 31 01:47:56.333174 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.333149    5650 cpu_manager.go:414] "containerMap: removing container" podUID="2946cec0-bd91-40ac-a095-6e174a3bf3b2" containerName="payload-processing"
Oct 31 01:47:56.333279 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.333175    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2946cec0-bd91-40ac-a095-6e174a3bf3b2" containerName="payload-processing"
Oct 31 01:47:56.333279 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.333209    5650 cpu_manager.go:414] "containerMap: removing container" podUID="3d94739d-327b-4ece-9b0a-bf38caeae06e" containerName="revoke-keys"
Oct 31 01:47:56.333279 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.333217    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d94739d-327b-4ece-9b0a-bf38caeae06e" containerName="revoke-keys"
Oct 31 01:47:56.333279 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.333258    5650 cpu_manager.go:414] "containerMap: removing container" podUID="80996937-b751-47a3-a515-0bef2d5e2327" containerName="istio-proxy"
Oct 31 01:47:56.333279 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.333266    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80996937-b751-47a3-a515-0bef2d5e2327" containerName="istio-proxy"
Oct 31 01:47:56.333547 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.333286    5650 cpu_manager.go:414] "containerMap: removing container" podUID="80996937-b751-47a3-a515-0bef2d5e2327" containerName="istio-proxy"
Oct 31 01:47:56.333547 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.333295    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80996937-b751-47a3-a515-0bef2d5e2327" containerName="istio-proxy"
Oct 31 01:47:56.333547 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.333328    5650 cpu_manager.go:414] "containerMap: removing container" podUID="d359baf9-7e49-4e89-afb1-7c763722c087" containerName="payload-pre-processing"
Oct 31 01:47:56.333547 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.333336    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d359baf9-7e49-4e89-afb1-7c763722c087" containerName="payload-pre-processing"
Oct 31 01:47:56.333547 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.333360    5650 cpu_manager.go:414] "containerMap: removing container" podUID="63fe4d8f-d251-4852-98e4-b20e54d74df1" containerName="istio-proxy"
Oct 31 01:47:56.333547 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.333368    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63fe4d8f-d251-4852-98e4-b20e54d74df1" containerName="istio-proxy"
Oct 31 01:47:56.333547 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.333535    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="d359baf9-7e49-4e89-afb1-7c763722c087" containerName="payload-pre-processing"
Oct 31 01:47:56.333898 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.333552    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="63fe4d8f-d251-4852-98e4-b20e54d74df1" containerName="istio-proxy"
Oct 31 01:47:56.333898 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.333561    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="2946cec0-bd91-40ac-a095-6e174a3bf3b2" containerName="payload-processing"
Oct 31 01:47:56.333898 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.333574    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d94739d-327b-4ece-9b0a-bf38caeae06e" containerName="revoke-keys"
Oct 31 01:47:56.333898 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.333589    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="80996937-b751-47a3-a515-0bef2d5e2327" containerName="istio-proxy"
Oct 31 01:47:56.339524 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.339504    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.354002 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.353460    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-03004a-openshift-default-dockercfg-xlzzf\""
Oct 31 01:47:56.357489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.355735    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"]
Oct 31 01:47:56.409376 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.409319    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-workload-certs\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.409592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.409467    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-envoy\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.409592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.409548    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-workload-socket\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.409739 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.409711    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-token\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.409988 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.409958    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27s4l\" (UniqueName: \"kubernetes.io/projected/35cccf52-b489-42d0-ae01-f3071bdc00ed-kube-api-access-27s4l\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.410080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.410057    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cccf52-b489-42d0-ae01-f3071bdc00ed-istiod-ca-cert\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.410202 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.410184    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-data\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.410356 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.410338    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-credential-socket\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.410426 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.410413    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-podinfo\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.511902 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.511859    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27s4l\" (UniqueName: \"kubernetes.io/projected/35cccf52-b489-42d0-ae01-f3071bdc00ed-kube-api-access-27s4l\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.512137 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.511927    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cccf52-b489-42d0-ae01-f3071bdc00ed-istiod-ca-cert\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.512137 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.511957    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-data\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.512137 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.512056    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-credential-socket\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.512318 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.512163    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-podinfo\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.512318 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.512205    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-workload-certs\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.512318 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.512292    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-envoy\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.512520 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.512316    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-workload-socket\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.512520 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.512392    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-token\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.514909 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.514828    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-credential-socket\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.516971 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.516931    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cccf52-b489-42d0-ae01-f3071bdc00ed-istiod-ca-cert\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.517735 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.517423    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-data\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.521436 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.521146    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-podinfo\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.521436 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.521399    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-workload-socket\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.558696 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.538383    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-token\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.558696 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.547266    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-envoy\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.558696 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.547585    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-workload-certs\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.588358 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.588261    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27s4l\" (UniqueName: \"kubernetes.io/projected/35cccf52-b489-42d0-ae01-f3071bdc00ed-kube-api-access-27s4l\") pod \"e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") " pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:56.660825 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:56.660779    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:57.013305 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:57.013274    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"]
Oct 31 01:47:57.014122 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:47:57.014090    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35cccf52_b489_42d0_ae01_f3071bdc00ed.slice/crio-8fd24fcb6ca495f1ca61e11d6ec89d9bf72344d26636527337afc2409bdde934 WatchSource:0}: Error finding container 8fd24fcb6ca495f1ca61e11d6ec89d9bf72344d26636527337afc2409bdde934: Status 404 returned error can't find the container with id 8fd24fcb6ca495f1ca61e11d6ec89d9bf72344d26636527337afc2409bdde934
Oct 31 01:47:57.016831 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:57.016796    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:47:57.016922 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:57.016870    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:47:57.016922 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:57.016900    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:47:57.423194 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:57.423098    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp" event={"ID":"35cccf52-b489-42d0-ae01-f3071bdc00ed","Type":"ContainerStarted","Data":"94491bea7057333a5d6b0d8796b2b74b2891ffaf050003e599b88fb4efb80fe7"}
Oct 31 01:47:57.423194 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:57.423166    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp" event={"ID":"35cccf52-b489-42d0-ae01-f3071bdc00ed","Type":"ContainerStarted","Data":"8fd24fcb6ca495f1ca61e11d6ec89d9bf72344d26636527337afc2409bdde934"}
Oct 31 01:47:57.452027 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:57.451965    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp" podStartSLOduration=1.4519471560000001 podStartE2EDuration="1.451947156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:57.449536694 +0000 UTC m=+1633.453250986" watchObservedRunningTime="2026-10-31 01:47:57.451947156 +0000 UTC m=+1633.455661447"
Oct 31 01:47:57.666078 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:57.661813    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:58.556199 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:47:58.556160    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:47:58.668426 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:58.668394    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:59.442941 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:59.442850    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:47:59.444261 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:47:59.444226    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:48:02.050466 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:02.049878    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2"]
Oct 31 01:48:02.050987 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:02.050953    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="80996937-b751-47a3-a515-0bef2d5e2327" containerName="istio-proxy"
Oct 31 01:48:02.057113 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:02.057078    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2"
Oct 31 01:48:02.064993 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:02.064955    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2"]
Oct 31 01:48:02.126572 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:02.116698    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9f0694\""
Oct 31 01:48:02.126572 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:02.116806    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9f0694-dockercfg-4qfhq\""
Oct 31 01:48:02.155550 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:02.130797    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghwd6\" (UniqueName: \"kubernetes.io/projected/c0ea3334-9ca3-4f52-94d4-7d52eaba45c2-kube-api-access-ghwd6\") pod \"payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2\" (UID: \"c0ea3334-9ca3-4f52-94d4-7d52eaba45c2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2"
Oct 31 01:48:02.155550 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:02.130894    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ea3334-9ca3-4f52-94d4-7d52eaba45c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2\" (UID: \"c0ea3334-9ca3-4f52-94d4-7d52eaba45c2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2"
Oct 31 01:48:02.236761 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:02.236706    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghwd6\" (UniqueName: \"kubernetes.io/projected/c0ea3334-9ca3-4f52-94d4-7d52eaba45c2-kube-api-access-ghwd6\") pod \"payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2\" (UID: \"c0ea3334-9ca3-4f52-94d4-7d52eaba45c2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2"
Oct 31 01:48:02.236964 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:02.236838    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ea3334-9ca3-4f52-94d4-7d52eaba45c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2\" (UID: \"c0ea3334-9ca3-4f52-94d4-7d52eaba45c2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2"
Oct 31 01:48:02.246021 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:02.243827    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ea3334-9ca3-4f52-94d4-7d52eaba45c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2\" (UID: \"c0ea3334-9ca3-4f52-94d4-7d52eaba45c2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2"
Oct 31 01:48:02.286588 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:02.284194    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghwd6\" (UniqueName: \"kubernetes.io/projected/c0ea3334-9ca3-4f52-94d4-7d52eaba45c2-kube-api-access-ghwd6\") pod \"payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2\" (UID: \"c0ea3334-9ca3-4f52-94d4-7d52eaba45c2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2"
Oct 31 01:48:02.376671 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:02.376560    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2"
Oct 31 01:48:02.664249 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:02.664212    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2"]
Oct 31 01:48:02.667269 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:48:02.667232    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ea3334_9ca3_4f52_94d4_7d52eaba45c2.slice/crio-991a6a8bdac3e78fd109e1689edbc3fda853946e9a8034db0bd6b613e2809c5f WatchSource:0}: Error finding container 991a6a8bdac3e78fd109e1689edbc3fda853946e9a8034db0bd6b613e2809c5f: Status 404 returned error can't find the container with id 991a6a8bdac3e78fd109e1689edbc3fda853946e9a8034db0bd6b613e2809c5f
Oct 31 01:48:03.592473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:03.591483    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2" event={"ID":"c0ea3334-9ca3-4f52-94d4-7d52eaba45c2","Type":"ContainerStarted","Data":"544ea1d850f9bc00018b4386edc558c68fec5bc8a9cd585afbde9e43ed450179"}
Oct 31 01:48:03.592473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:03.591588    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2" event={"ID":"c0ea3334-9ca3-4f52-94d4-7d52eaba45c2","Type":"ContainerStarted","Data":"991a6a8bdac3e78fd109e1689edbc3fda853946e9a8034db0bd6b613e2809c5f"}
Oct 31 01:48:03.604469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:03.602542    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2"
Oct 31 01:48:03.647803 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:03.647421    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2" podStartSLOduration=1.647397727 podStartE2EDuration="1.647397727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:03.631300557 +0000 UTC m=+1639.635014851" watchObservedRunningTime="2026-10-31 01:48:03.647397727 +0000 UTC m=+1639.651112020"
Oct 31 01:48:15.676471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:15.670661    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:48:15.699631 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:15.699582    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:48:15.839477 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:15.826412    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Oct 31 01:48:15.839477 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:15.830537    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:48:15.839477 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:15.835567    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:48:15.876485 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:15.852768    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2"
Oct 31 01:48:15.887481 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:15.886628    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:48:15.950531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:15.949582    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 01:48:15.981690 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:15.975589    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55" podUID="96bb0499-5edf-49b8-8391-f7948587ff19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:48:15.981690 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:15.976421    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 01:48:16.703991 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:16.703932    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:48:16.705884 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:16.705841    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:48:16.760483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:16.758096    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:48:16.760483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:16.758176    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:48:16.793473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:16.780577    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded"
Oct 31 01:48:16.833497 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:16.831815    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:48:16.833497 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:16.832233    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Oct 31 01:48:16.833497 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:16.832823    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:48:16.833497 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:16.832884    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:48:16.838040 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:16.836533    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:48:16.838040 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:16.837234    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:48:16.977583 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:16.974610    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 01:48:16.977583 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:16.976014    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:48:17.115954 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:17.115924    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:48:17.181523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:17.179504    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 01:48:17.181523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:17.181225    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55" podUID="96bb0499-5edf-49b8-8391-f7948587ff19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded"
Oct 31 01:48:17.181994 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:17.181952    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 01:48:17.187076 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:17.187049    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:48:17.187438 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:17.187394    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:48:17.189969 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:17.189945    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:48:17.284724 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:17.266949    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:48:17.287765 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:17.286904    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:48:17.308940 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:17.308772    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:48:17.324466 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:17.316304    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:48:17.324466 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:17.316383    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:48:17.337835 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:17.337806    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:48:17.356225 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:17.356195    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:48:17.372560 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:17.372530    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:48:20.436518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:20.436483    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2"]
Oct 31 01:48:25.437177 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:25.437132    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2" podUID="c0ea3334-9ca3-4f52-94d4-7d52eaba45c2" containerName="payload-pre-processing" containerID="cri-o://544ea1d850f9bc00018b4386edc558c68fec5bc8a9cd585afbde9e43ed450179" gracePeriod=25
Oct 31 01:48:25.609145 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:48:25.609113    5650 logging.go:55] [core] [Channel #817 SubChannel #818]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.108:50052", ServerName: "10.134.0.108:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.108:50052: connect: connection refused"
Oct 31 01:48:25.747235 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:25.747199    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2"
Oct 31 01:48:25.807312 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:25.807273    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ea3334-9ca3-4f52-94d4-7d52eaba45c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ea3334-9ca3-4f52-94d4-7d52eaba45c2-plugins-config-volume\") pod \"c0ea3334-9ca3-4f52-94d4-7d52eaba45c2\" (UID: \"c0ea3334-9ca3-4f52-94d4-7d52eaba45c2\") "
Oct 31 01:48:25.807664 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:25.807642    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ea3334-9ca3-4f52-94d4-7d52eaba45c2-kube-api-access-ghwd6\" (UniqueName: \"kubernetes.io/projected/c0ea3334-9ca3-4f52-94d4-7d52eaba45c2-kube-api-access-ghwd6\") pod \"c0ea3334-9ca3-4f52-94d4-7d52eaba45c2\" (UID: \"c0ea3334-9ca3-4f52-94d4-7d52eaba45c2\") "
Oct 31 01:48:25.807885 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:25.807859    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ea3334-9ca3-4f52-94d4-7d52eaba45c2-plugins-config-volume" pod "c0ea3334-9ca3-4f52-94d4-7d52eaba45c2" (UID: "c0ea3334-9ca3-4f52-94d4-7d52eaba45c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:48:25.808736 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:25.808706    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ea3334-9ca3-4f52-94d4-7d52eaba45c2-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:25.810728 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:25.810690    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ea3334-9ca3-4f52-94d4-7d52eaba45c2-kube-api-access-ghwd6" pod "c0ea3334-9ca3-4f52-94d4-7d52eaba45c2" (UID: "c0ea3334-9ca3-4f52-94d4-7d52eaba45c2"). InnerVolumeSpecName "kube-api-access-ghwd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:48:25.910777 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:25.910739    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghwd6\" (UniqueName: \"kubernetes.io/projected/c0ea3334-9ca3-4f52-94d4-7d52eaba45c2-kube-api-access-ghwd6\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:26.313593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:26.313554    5650 generic.go:356] "Generic (PLEG): container finished" podID="c0ea3334-9ca3-4f52-94d4-7d52eaba45c2" containerID="544ea1d850f9bc00018b4386edc558c68fec5bc8a9cd585afbde9e43ed450179" exitCode=0
Oct 31 01:48:26.313593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:26.313599    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2"
Oct 31 01:48:26.313892 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:26.313637    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2" event={"ID":"c0ea3334-9ca3-4f52-94d4-7d52eaba45c2","Type":"ContainerDied","Data":"544ea1d850f9bc00018b4386edc558c68fec5bc8a9cd585afbde9e43ed450179"}
Oct 31 01:48:26.313892 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:26.313697    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2" event={"ID":"c0ea3334-9ca3-4f52-94d4-7d52eaba45c2","Type":"ContainerDied","Data":"991a6a8bdac3e78fd109e1689edbc3fda853946e9a8034db0bd6b613e2809c5f"}
Oct 31 01:48:26.313892 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:26.313724    5650 scope.go:122] "RemoveContainer" containerID="544ea1d850f9bc00018b4386edc558c68fec5bc8a9cd585afbde9e43ed450179"
Oct 31 01:48:26.336549 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:26.336522    5650 scope.go:122] "RemoveContainer" containerID="544ea1d850f9bc00018b4386edc558c68fec5bc8a9cd585afbde9e43ed450179"
Oct 31 01:48:26.336939 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:48:26.336915    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"544ea1d850f9bc00018b4386edc558c68fec5bc8a9cd585afbde9e43ed450179\": container with ID starting with 544ea1d850f9bc00018b4386edc558c68fec5bc8a9cd585afbde9e43ed450179 not found: ID does not exist" containerID="544ea1d850f9bc00018b4386edc558c68fec5bc8a9cd585afbde9e43ed450179"
Oct 31 01:48:26.337014 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:26.336951    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"544ea1d850f9bc00018b4386edc558c68fec5bc8a9cd585afbde9e43ed450179"} err="failed to get container status \"544ea1d850f9bc00018b4386edc558c68fec5bc8a9cd585afbde9e43ed450179\": rpc error: code = NotFound desc = could not find container \"544ea1d850f9bc00018b4386edc558c68fec5bc8a9cd585afbde9e43ed450179\": container with ID starting with 544ea1d850f9bc00018b4386edc558c68fec5bc8a9cd585afbde9e43ed450179 not found: ID does not exist"
Oct 31 01:48:26.349885 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:26.349836    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2"]
Oct 31 01:48:26.357718 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:26.357679    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2"]
Oct 31 01:48:26.608312 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:26.608268    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9f0694-747879f447-5mrn2" podUID="c0ea3334-9ca3-4f52-94d4-7d52eaba45c2" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.108:50052\" within 1s: context deadline exceeded"
Oct 31 01:48:28.119247 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:28.119204    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0ea3334-9ca3-4f52-94d4-7d52eaba45c2" path="/var/lib/kubelet/pods/c0ea3334-9ca3-4f52-94d4-7d52eaba45c2/volumes"
Oct 31 01:48:28.417045 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:28.416951    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"]
Oct 31 01:48:28.554164 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:48:28.554125    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:48:43.439826 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:43.439767    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55" podUID="96bb0499-5edf-49b8-8391-f7948587ff19" containerName="main" containerID="cri-o://e701c9b2c22f7cf7d678bc0b0e5a19ce65c1f20c2491983596d5f82d910ac48b" gracePeriod=45
Oct 31 01:48:43.631938 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:43.631904    5650 generic.go:356] "Generic (PLEG): container finished" podID="96bb0499-5edf-49b8-8391-f7948587ff19" containerID="e701c9b2c22f7cf7d678bc0b0e5a19ce65c1f20c2491983596d5f82d910ac48b" exitCode=0
Oct 31 01:48:43.632096 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:43.631975    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55" event={"ID":"96bb0499-5edf-49b8-8391-f7948587ff19","Type":"ContainerDied","Data":"e701c9b2c22f7cf7d678bc0b0e5a19ce65c1f20c2491983596d5f82d910ac48b"}
Oct 31 01:48:43.828174 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:43.828136    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:48:43.849567 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:43.849531    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-home\" (UniqueName: \"kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-home\") pod \"96bb0499-5edf-49b8-8391-f7948587ff19\" (UID: \"96bb0499-5edf-49b8-8391-f7948587ff19\") "
Oct 31 01:48:43.849717 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:43.849669    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-dshm\" (UniqueName: \"kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-dshm\") pod \"96bb0499-5edf-49b8-8391-f7948587ff19\" (UID: \"96bb0499-5edf-49b8-8391-f7948587ff19\") "
Oct 31 01:48:43.849761 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:43.849745    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-tmp-dir\") pod \"96bb0499-5edf-49b8-8391-f7948587ff19\" (UID: \"96bb0499-5edf-49b8-8391-f7948587ff19\") "
Oct 31 01:48:43.849822 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:43.849800    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-home" pod "96bb0499-5edf-49b8-8391-f7948587ff19" (UID: "96bb0499-5edf-49b8-8391-f7948587ff19"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:48:43.849890 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:43.849877    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-model-cache\") pod \"96bb0499-5edf-49b8-8391-f7948587ff19\" (UID: \"96bb0499-5edf-49b8-8391-f7948587ff19\") "
Oct 31 01:48:43.849926 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:43.849906    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96bb0499-5edf-49b8-8391-f7948587ff19-kube-api-access-jmk74\" (UniqueName: \"kubernetes.io/projected/96bb0499-5edf-49b8-8391-f7948587ff19-kube-api-access-jmk74\") pod \"96bb0499-5edf-49b8-8391-f7948587ff19\" (UID: \"96bb0499-5edf-49b8-8391-f7948587ff19\") "
Oct 31 01:48:43.850010 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:43.849996    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96bb0499-5edf-49b8-8391-f7948587ff19-tls-certs\" (UniqueName: \"kubernetes.io/secret/96bb0499-5edf-49b8-8391-f7948587ff19-tls-certs\") pod \"96bb0499-5edf-49b8-8391-f7948587ff19\" (UID: \"96bb0499-5edf-49b8-8391-f7948587ff19\") "
Oct 31 01:48:43.850057 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:43.850008    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-tmp-dir" pod "96bb0499-5edf-49b8-8391-f7948587ff19" (UID: "96bb0499-5edf-49b8-8391-f7948587ff19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:48:43.850118 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:43.850104    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-model-cache" pod "96bb0499-5edf-49b8-8391-f7948587ff19" (UID: "96bb0499-5edf-49b8-8391-f7948587ff19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:48:43.850639 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:43.850618    5650 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-model-cache\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:43.850639 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:43.850636    5650 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-home\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:43.850836 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:43.850645    5650 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-tmp-dir\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:43.852014 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:43.851987    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-dshm" pod "96bb0499-5edf-49b8-8391-f7948587ff19" (UID: "96bb0499-5edf-49b8-8391-f7948587ff19"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:48:43.852127 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:43.852042    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96bb0499-5edf-49b8-8391-f7948587ff19-tls-certs" pod "96bb0499-5edf-49b8-8391-f7948587ff19" (UID: "96bb0499-5edf-49b8-8391-f7948587ff19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:48:43.852127 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:43.852047    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96bb0499-5edf-49b8-8391-f7948587ff19-kube-api-access-jmk74" pod "96bb0499-5edf-49b8-8391-f7948587ff19" (UID: "96bb0499-5edf-49b8-8391-f7948587ff19"). InnerVolumeSpecName "kube-api-access-jmk74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:48:43.952174 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:43.952137    5650 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96bb0499-5edf-49b8-8391-f7948587ff19-dshm\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:43.952174 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:43.952168    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmk74\" (UniqueName: \"kubernetes.io/projected/96bb0499-5edf-49b8-8391-f7948587ff19-kube-api-access-jmk74\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:43.952174 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:43.952179    5650 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96bb0499-5edf-49b8-8391-f7948587ff19-tls-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:48:44.443995 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:44.443890    5650 patch_prober.go:29] interesting pod/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:48:44.443995 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:44.443957    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp" podUID="35cccf52-b489-42d0-ae01-f3071bdc00ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 01:48:44.537030 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:44.536996    5650 scope.go:122] "RemoveContainer" containerID="85f513b2a91f54dc1286917b41aa6951f7e2a48fa1790eacc63a9cccdb5ad8c6"
Oct 31 01:48:44.641086 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:44.641051    5650 generic.go:356] "Generic (PLEG): container finished" podID="35cccf52-b489-42d0-ae01-f3071bdc00ed" containerID="94491bea7057333a5d6b0d8796b2b74b2891ffaf050003e599b88fb4efb80fe7" exitCode=0
Oct 31 01:48:44.641284 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:44.641133    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp" event={"ID":"35cccf52-b489-42d0-ae01-f3071bdc00ed","Type":"ContainerDied","Data":"94491bea7057333a5d6b0d8796b2b74b2891ffaf050003e599b88fb4efb80fe7"}
Oct 31 01:48:44.641754 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:44.641732    5650 scope.go:122] "RemoveContainer" containerID="94491bea7057333a5d6b0d8796b2b74b2891ffaf050003e599b88fb4efb80fe7"
Oct 31 01:48:44.642555 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:44.642540    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"
Oct 31 01:48:44.642637 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:44.642541    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55" event={"ID":"96bb0499-5edf-49b8-8391-f7948587ff19","Type":"ContainerDied","Data":"c7677eda7957d9a68a68b8ca491d6b16e13bc3d8e9ab75a79f5c747afa380daf"}
Oct 31 01:48:44.642707 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:44.642660    5650 scope.go:122] "RemoveContainer" containerID="e701c9b2c22f7cf7d678bc0b0e5a19ce65c1f20c2491983596d5f82d910ac48b"
Oct 31 01:48:44.643593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:44.643483    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:48:44.643593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:44.643566    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:48:44.643708 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:44.643613    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:48:44.686399 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:44.686365    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"]
Oct 31 01:48:44.694405 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:44.694337    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-cb5756cb/e2e-lifecycle-model-cb5756cb-kserve-6985dc9fb9-zmx55"]
Oct 31 01:48:45.659391 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:45.659351    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp" event={"ID":"35cccf52-b489-42d0-ae01-f3071bdc00ed","Type":"ContainerStarted","Data":"75e24e03ff8c864169d6bebdd6ef912ef5e79bc626feea5471a72e6b2c1814cd"}
Oct 31 01:48:45.662110 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:45.662079    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:48:45.662300 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:45.662260    5650 patch_prober.go:29] interesting pod/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:48:45.662380 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:45.662317    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp" podUID="35cccf52-b489-42d0-ae01-f3071bdc00ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 01:48:46.131257 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:46.131221    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96bb0499-5edf-49b8-8391-f7948587ff19" path="/var/lib/kubelet/pods/96bb0499-5edf-49b8-8391-f7948587ff19/volumes"
Oct 31 01:48:46.662703 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:46.662658    5650 patch_prober.go:29] interesting pod/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:48:46.663124 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:46.662730    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp" podUID="35cccf52-b489-42d0-ae01-f3071bdc00ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 01:48:47.662268 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:47.662234    5650 patch_prober.go:29] interesting pod/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:48:47.662490 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:47.662286    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp" podUID="35cccf52-b489-42d0-ae01-f3071bdc00ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 01:48:47.894560 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:47.894508    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"]
Oct 31 01:48:47.895326 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:47.895306    5650 cpu_manager.go:414] "containerMap: removing container" podUID="c0ea3334-9ca3-4f52-94d4-7d52eaba45c2" containerName="payload-pre-processing"
Oct 31 01:48:47.943760 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:47.895514    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ea3334-9ca3-4f52-94d4-7d52eaba45c2" containerName="payload-pre-processing"
Oct 31 01:48:47.943937 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:47.943761    5650 cpu_manager.go:414] "containerMap: removing container" podUID="96bb0499-5edf-49b8-8391-f7948587ff19" containerName="main"
Oct 31 01:48:47.943937 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:47.943777    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96bb0499-5edf-49b8-8391-f7948587ff19" containerName="main"
Oct 31 01:48:47.944587 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:47.944434    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="96bb0499-5edf-49b8-8391-f7948587ff19" containerName="main"
Oct 31 01:48:47.944714 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:47.944679    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ea3334-9ca3-4f52-94d4-7d52eaba45c2" containerName="payload-pre-processing"
Oct 31 01:48:47.951917 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:47.951873    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"]
Oct 31 01:48:47.952073 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:47.951971    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:47.963561 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:47.963518    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-cbbaec-openshift-default-dockercfg-xvh4c\""
Oct 31 01:48:48.068834 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.068677    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-workload-socket\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.069151 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.069102    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-data\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.069361 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.069345    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-credential-socket\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.069571 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.069555    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-podinfo\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.069704 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.069693    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v845r\" (UniqueName: \"kubernetes.io/projected/868f691d-84c7-4301-b42f-3a314ffcdb0b-kube-api-access-v845r\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.069888 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.069876    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-workload-certs\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.070064 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.070040    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-token\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.070219 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.070194    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/868f691d-84c7-4301-b42f-3a314ffcdb0b-istiod-ca-cert\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.070342 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.070332    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-envoy\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.172280 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.172241    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-credential-socket\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.172511 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.172375    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-podinfo\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.172511 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.172420    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v845r\" (UniqueName: \"kubernetes.io/projected/868f691d-84c7-4301-b42f-3a314ffcdb0b-kube-api-access-v845r\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.172706 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.172519    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-workload-certs\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.172706 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.172594    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-token\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.172706 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.172649    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/868f691d-84c7-4301-b42f-3a314ffcdb0b-istiod-ca-cert\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.172706 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.172691    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-envoy\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.172880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.172810    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-workload-socket\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.172880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.172861    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-data\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.173415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.173386    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-data\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.173723 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.173702    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-workload-socket\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.173808 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.173763    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/868f691d-84c7-4301-b42f-3a314ffcdb0b-istiod-ca-cert\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.174623 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.174603    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-workload-certs\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.174818 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.174765    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-credential-socket\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.176983 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.176919    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-envoy\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.212984 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.209549    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v845r\" (UniqueName: \"kubernetes.io/projected/868f691d-84c7-4301-b42f-3a314ffcdb0b-kube-api-access-v845r\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.250472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.248617    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-podinfo\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.268468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.257225    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-token\") pod \"e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") " pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.275239 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.271518    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:48.662532 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.662497    5650 patch_prober.go:29] interesting pod/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:48:48.662712 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.662566    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp" podUID="35cccf52-b489-42d0-ae01-f3071bdc00ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 01:48:48.765021 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.764934    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"]
Oct 31 01:48:48.766726 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:48:48.766688    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod868f691d_84c7_4301_b42f_3a314ffcdb0b.slice/crio-9f61bade8b1c457a85a57a97864361d97c10ad4042b91f1fd0d0ff760e5f2bf4 WatchSource:0}: Error finding container 9f61bade8b1c457a85a57a97864361d97c10ad4042b91f1fd0d0ff760e5f2bf4: Status 404 returned error can't find the container with id 9f61bade8b1c457a85a57a97864361d97c10ad4042b91f1fd0d0ff760e5f2bf4
Oct 31 01:48:48.769583 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.769543    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:48:48.769715 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.769626    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:48:48.769715 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:48.769661    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:48:49.662110 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:49.662076    5650 patch_prober.go:29] interesting pod/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:48:49.662565 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:49.662130    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp" podUID="35cccf52-b489-42d0-ae01-f3071bdc00ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 01:48:49.697506 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:49.697441    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2" event={"ID":"868f691d-84c7-4301-b42f-3a314ffcdb0b","Type":"ContainerStarted","Data":"7d2c5aa798f3989219b918d5ba62db13055c9cb10d601bba4a8940b89640e904"}
Oct 31 01:48:49.697697 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:49.697527    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2" event={"ID":"868f691d-84c7-4301-b42f-3a314ffcdb0b","Type":"ContainerStarted","Data":"9f61bade8b1c457a85a57a97864361d97c10ad4042b91f1fd0d0ff760e5f2bf4"}
Oct 31 01:48:49.729367 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:49.729306    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2" podStartSLOduration=2.729286862 podStartE2EDuration="2.729286862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:49.725032672 +0000 UTC m=+1685.728746979" watchObservedRunningTime="2026-10-31 01:48:49.729286862 +0000 UTC m=+1685.733001152"
Oct 31 01:48:50.272098 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:50.272066    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:50.278745 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:50.278714    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:50.662108 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:50.662029    5650 patch_prober.go:29] interesting pod/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:48:50.662108 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:50.662081    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp" podUID="35cccf52-b489-42d0-ae01-f3071bdc00ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 01:48:50.705339 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:50.705301    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:50.706396 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:50.706377    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:48:51.669733 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:51.669700    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:48:51.670267 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:51.670244    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:48:51.671090 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:48:51.671067    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:48:58.560819 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:48:58.559911    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:49:12.414476 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:12.414395    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m"]
Oct 31 01:49:12.418800 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:12.418781    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m"
Oct 31 01:49:12.425643 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:12.425616    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-073e24\""
Oct 31 01:49:12.425782 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:12.425689    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-073e24-dockercfg-djhw4\""
Oct 31 01:49:12.429238 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:12.429214    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m"]
Oct 31 01:49:12.461393 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:12.461350    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g"]
Oct 31 01:49:12.465952 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:12.465926    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g"
Oct 31 01:49:12.477520 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:12.477487    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g"]
Oct 31 01:49:12.513277 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:12.513243    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ssxk\" (UniqueName: \"kubernetes.io/projected/1571b2ff-b45f-44ae-bc9c-4043fd61a2b0-kube-api-access-6ssxk\") pod \"payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m\" (UID: \"1571b2ff-b45f-44ae-bc9c-4043fd61a2b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m"
Oct 31 01:49:12.513463 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:12.513288    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1571b2ff-b45f-44ae-bc9c-4043fd61a2b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m\" (UID: \"1571b2ff-b45f-44ae-bc9c-4043fd61a2b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m"
Oct 31 01:49:12.513514 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:12.513489    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvw2r\" (UniqueName: \"kubernetes.io/projected/06974ced-b6e7-4dc3-9003-1a1d6d8acd12-kube-api-access-fvw2r\") pod \"payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g\" (UID: \"06974ced-b6e7-4dc3-9003-1a1d6d8acd12\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g"
Oct 31 01:49:12.513550 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:12.513534    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06974ced-b6e7-4dc3-9003-1a1d6d8acd12-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g\" (UID: \"06974ced-b6e7-4dc3-9003-1a1d6d8acd12\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g"
Oct 31 01:49:12.624976 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:12.621568    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ssxk\" (UniqueName: \"kubernetes.io/projected/1571b2ff-b45f-44ae-bc9c-4043fd61a2b0-kube-api-access-6ssxk\") pod \"payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m\" (UID: \"1571b2ff-b45f-44ae-bc9c-4043fd61a2b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m"
Oct 31 01:49:12.624976 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:12.621632    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1571b2ff-b45f-44ae-bc9c-4043fd61a2b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m\" (UID: \"1571b2ff-b45f-44ae-bc9c-4043fd61a2b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m"
Oct 31 01:49:12.624976 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:12.621792    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvw2r\" (UniqueName: \"kubernetes.io/projected/06974ced-b6e7-4dc3-9003-1a1d6d8acd12-kube-api-access-fvw2r\") pod \"payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g\" (UID: \"06974ced-b6e7-4dc3-9003-1a1d6d8acd12\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g"
Oct 31 01:49:12.624976 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:12.621837    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06974ced-b6e7-4dc3-9003-1a1d6d8acd12-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g\" (UID: \"06974ced-b6e7-4dc3-9003-1a1d6d8acd12\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g"
Oct 31 01:49:12.624976 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:12.622897    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06974ced-b6e7-4dc3-9003-1a1d6d8acd12-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g\" (UID: \"06974ced-b6e7-4dc3-9003-1a1d6d8acd12\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g"
Oct 31 01:49:12.632477 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:12.627806    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1571b2ff-b45f-44ae-bc9c-4043fd61a2b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m\" (UID: \"1571b2ff-b45f-44ae-bc9c-4043fd61a2b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m"
Oct 31 01:49:12.691514 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:12.689691    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ssxk\" (UniqueName: \"kubernetes.io/projected/1571b2ff-b45f-44ae-bc9c-4043fd61a2b0-kube-api-access-6ssxk\") pod \"payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m\" (UID: \"1571b2ff-b45f-44ae-bc9c-4043fd61a2b0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m"
Oct 31 01:49:12.744514 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:12.717080    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvw2r\" (UniqueName: \"kubernetes.io/projected/06974ced-b6e7-4dc3-9003-1a1d6d8acd12-kube-api-access-fvw2r\") pod \"payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g\" (UID: \"06974ced-b6e7-4dc3-9003-1a1d6d8acd12\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g"
Oct 31 01:49:12.744863 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:12.736367    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m"
Oct 31 01:49:12.828725 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:12.828668    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g"
Oct 31 01:49:13.178008 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:49:13.177045    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06974ced_b6e7_4dc3_9003_1a1d6d8acd12.slice/crio-4e90228c81e4bb44a9b9442b97bb36d2070efd5a6ae381a633fbe7f9411fd281 WatchSource:0}: Error finding container 4e90228c81e4bb44a9b9442b97bb36d2070efd5a6ae381a633fbe7f9411fd281: Status 404 returned error can't find the container with id 4e90228c81e4bb44a9b9442b97bb36d2070efd5a6ae381a633fbe7f9411fd281
Oct 31 01:49:13.178709 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:13.178682    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g"]
Oct 31 01:49:13.181185 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:49:13.181033    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1571b2ff_b45f_44ae_bc9c_4043fd61a2b0.slice/crio-0ae5b81a4e4f7deadc66e4a634c8251feedd174f5ed291236a3aa4897e9a607f WatchSource:0}: Error finding container 0ae5b81a4e4f7deadc66e4a634c8251feedd174f5ed291236a3aa4897e9a607f: Status 404 returned error can't find the container with id 0ae5b81a4e4f7deadc66e4a634c8251feedd174f5ed291236a3aa4897e9a607f
Oct 31 01:49:13.183492 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:13.183431    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m"]
Oct 31 01:49:14.100553 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:14.100513    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g" event={"ID":"06974ced-b6e7-4dc3-9003-1a1d6d8acd12","Type":"ContainerStarted","Data":"3b3d30a59149f06412d7a5a6884f664632444e7a6fa337778aaaeb0de44edd0e"}
Oct 31 01:49:14.101050 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:14.100573    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g" event={"ID":"06974ced-b6e7-4dc3-9003-1a1d6d8acd12","Type":"ContainerStarted","Data":"4e90228c81e4bb44a9b9442b97bb36d2070efd5a6ae381a633fbe7f9411fd281"}
Oct 31 01:49:14.101050 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:14.100603    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g"
Oct 31 01:49:14.102391 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:14.102361    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m" event={"ID":"1571b2ff-b45f-44ae-bc9c-4043fd61a2b0","Type":"ContainerStarted","Data":"bc73861d6e2443dd9628f546be4b87a7907ae2bcfd7c21082b31c32b895b76e9"}
Oct 31 01:49:14.102554 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:14.102407    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m" event={"ID":"1571b2ff-b45f-44ae-bc9c-4043fd61a2b0","Type":"ContainerStarted","Data":"0ae5b81a4e4f7deadc66e4a634c8251feedd174f5ed291236a3aa4897e9a607f"}
Oct 31 01:49:14.102554 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:14.102470    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m"
Oct 31 01:49:14.124002 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:14.123946    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g" podStartSLOduration=2.123931256 podStartE2EDuration="2.123931256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:14.120860893 +0000 UTC m=+1710.124575184" watchObservedRunningTime="2026-10-31 01:49:14.123931256 +0000 UTC m=+1710.127645550"
Oct 31 01:49:14.151553 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:14.151506    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m" podStartSLOduration=2.151491341 podStartE2EDuration="2.151491341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:14.148638866 +0000 UTC m=+1710.152353157" watchObservedRunningTime="2026-10-31 01:49:14.151491341 +0000 UTC m=+1710.155205652"
Oct 31 01:49:14.616590 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:14.616557    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"]
Oct 31 01:49:14.616891 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:14.616775    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd" podUID="1357b02f-b722-40ac-8ec2-35f5626d8347" containerName="istio-proxy" containerID="cri-o://bb5d261e53cbb81da581b0efa07149075735abe5bb86a42ee0db05621708a1b9" gracePeriod=30
Oct 31 01:49:14.779478 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:14.778514    5650 patch_prober.go:29] interesting pod/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:49:14.779478 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:14.778581    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd" podUID="1357b02f-b722-40ac-8ec2-35f5626d8347" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:49:20.034825 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.034787    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:49:20.176559 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.176519    5650 generic.go:356] "Generic (PLEG): container finished" podID="1357b02f-b722-40ac-8ec2-35f5626d8347" containerID="bb5d261e53cbb81da581b0efa07149075735abe5bb86a42ee0db05621708a1b9" exitCode=0
Oct 31 01:49:20.176778 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.176752    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd" event={"ID":"1357b02f-b722-40ac-8ec2-35f5626d8347","Type":"ContainerDied","Data":"bb5d261e53cbb81da581b0efa07149075735abe5bb86a42ee0db05621708a1b9"}
Oct 31 01:49:20.176863 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.176822    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd" event={"ID":"1357b02f-b722-40ac-8ec2-35f5626d8347","Type":"ContainerDied","Data":"e4bf28fcb669a2595cb706a558e89e3efa6c6724255a200b367e837475e27497"}
Oct 31 01:49:20.176863 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.176852    5650 scope.go:122] "RemoveContainer" containerID="bb5d261e53cbb81da581b0efa07149075735abe5bb86a42ee0db05621708a1b9"
Oct 31 01:49:20.177052 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.177032    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"
Oct 31 01:49:20.229467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.217941    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-workload-socket\") pod \"1357b02f-b722-40ac-8ec2-35f5626d8347\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") "
Oct 31 01:49:20.229467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.218050    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-data\") pod \"1357b02f-b722-40ac-8ec2-35f5626d8347\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") "
Oct 31 01:49:20.265326 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.262601    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-workload-certs\") pod \"1357b02f-b722-40ac-8ec2-35f5626d8347\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") "
Oct 31 01:49:20.265326 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.262855    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-envoy\") pod \"1357b02f-b722-40ac-8ec2-35f5626d8347\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") "
Oct 31 01:49:20.265326 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.262915    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1357b02f-b722-40ac-8ec2-35f5626d8347-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1357b02f-b722-40ac-8ec2-35f5626d8347-istiod-ca-cert\") pod \"1357b02f-b722-40ac-8ec2-35f5626d8347\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") "
Oct 31 01:49:20.265326 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.262962    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-token\" (UniqueName: \"kubernetes.io/projected/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-token\") pod \"1357b02f-b722-40ac-8ec2-35f5626d8347\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") "
Oct 31 01:49:20.265326 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.263016    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-podinfo\") pod \"1357b02f-b722-40ac-8ec2-35f5626d8347\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") "
Oct 31 01:49:20.265326 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.263077    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1357b02f-b722-40ac-8ec2-35f5626d8347-kube-api-access-92f6d\" (UniqueName: \"kubernetes.io/projected/1357b02f-b722-40ac-8ec2-35f5626d8347-kube-api-access-92f6d\") pod \"1357b02f-b722-40ac-8ec2-35f5626d8347\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") "
Oct 31 01:49:20.265326 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.263164    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-credential-socket\") pod \"1357b02f-b722-40ac-8ec2-35f5626d8347\" (UID: \"1357b02f-b722-40ac-8ec2-35f5626d8347\") "
Oct 31 01:49:20.288153 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.288123    5650 scope.go:122] "RemoveContainer" containerID="97cb4f814576543ee642a0a0a5995e2c4c11197cd4b9e66d10192b234e51f864"
Oct 31 01:49:20.288520 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.288429    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-workload-socket" pod "1357b02f-b722-40ac-8ec2-35f5626d8347" (UID: "1357b02f-b722-40ac-8ec2-35f5626d8347"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:49:20.292631 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.292601    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-token" pod "1357b02f-b722-40ac-8ec2-35f5626d8347" (UID: "1357b02f-b722-40ac-8ec2-35f5626d8347"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:49:20.293242 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.293218    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-workload-certs" pod "1357b02f-b722-40ac-8ec2-35f5626d8347" (UID: "1357b02f-b722-40ac-8ec2-35f5626d8347"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:49:20.293407 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.293391    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-data" pod "1357b02f-b722-40ac-8ec2-35f5626d8347" (UID: "1357b02f-b722-40ac-8ec2-35f5626d8347"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:49:20.295392 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.295365    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-credential-socket" pod "1357b02f-b722-40ac-8ec2-35f5626d8347" (UID: "1357b02f-b722-40ac-8ec2-35f5626d8347"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:49:20.295641 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.295616    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-podinfo" pod "1357b02f-b722-40ac-8ec2-35f5626d8347" (UID: "1357b02f-b722-40ac-8ec2-35f5626d8347"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:49:20.303476 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.296241    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1357b02f-b722-40ac-8ec2-35f5626d8347-istiod-ca-cert" pod "1357b02f-b722-40ac-8ec2-35f5626d8347" (UID: "1357b02f-b722-40ac-8ec2-35f5626d8347"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:49:20.359784 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.359742    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-envoy" pod "1357b02f-b722-40ac-8ec2-35f5626d8347" (UID: "1357b02f-b722-40ac-8ec2-35f5626d8347"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:49:20.361647 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.361582    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1357b02f-b722-40ac-8ec2-35f5626d8347-kube-api-access-92f6d" pod "1357b02f-b722-40ac-8ec2-35f5626d8347" (UID: "1357b02f-b722-40ac-8ec2-35f5626d8347"). InnerVolumeSpecName "kube-api-access-92f6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:49:20.378575 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.377954    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:20.378575 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.377993    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1357b02f-b722-40ac-8ec2-35f5626d8347-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:20.378575 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.378017    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:20.378575 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.378040    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:20.378575 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.378055    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92f6d\" (UniqueName: \"kubernetes.io/projected/1357b02f-b722-40ac-8ec2-35f5626d8347-kube-api-access-92f6d\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:20.378575 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.378076    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:20.378575 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.378096    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:20.378575 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.378109    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:20.378575 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.378114    5650 scope.go:122] "RemoveContainer" containerID="bb5d261e53cbb81da581b0efa07149075735abe5bb86a42ee0db05621708a1b9"
Oct 31 01:49:20.378575 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.378122    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1357b02f-b722-40ac-8ec2-35f5626d8347-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:20.380834 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:49:20.380668    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb5d261e53cbb81da581b0efa07149075735abe5bb86a42ee0db05621708a1b9\": container with ID starting with bb5d261e53cbb81da581b0efa07149075735abe5bb86a42ee0db05621708a1b9 not found: ID does not exist" containerID="bb5d261e53cbb81da581b0efa07149075735abe5bb86a42ee0db05621708a1b9"
Oct 31 01:49:20.380834 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.380709    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb5d261e53cbb81da581b0efa07149075735abe5bb86a42ee0db05621708a1b9"} err="failed to get container status \"bb5d261e53cbb81da581b0efa07149075735abe5bb86a42ee0db05621708a1b9\": rpc error: code = NotFound desc = could not find container \"bb5d261e53cbb81da581b0efa07149075735abe5bb86a42ee0db05621708a1b9\": container with ID starting with bb5d261e53cbb81da581b0efa07149075735abe5bb86a42ee0db05621708a1b9 not found: ID does not exist"
Oct 31 01:49:20.380834 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.380735    5650 scope.go:122] "RemoveContainer" containerID="97cb4f814576543ee642a0a0a5995e2c4c11197cd4b9e66d10192b234e51f864"
Oct 31 01:49:20.381233 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:49:20.381134    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97cb4f814576543ee642a0a0a5995e2c4c11197cd4b9e66d10192b234e51f864\": container with ID starting with 97cb4f814576543ee642a0a0a5995e2c4c11197cd4b9e66d10192b234e51f864 not found: ID does not exist" containerID="97cb4f814576543ee642a0a0a5995e2c4c11197cd4b9e66d10192b234e51f864"
Oct 31 01:49:20.381233 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.381167    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97cb4f814576543ee642a0a0a5995e2c4c11197cd4b9e66d10192b234e51f864"} err="failed to get container status \"97cb4f814576543ee642a0a0a5995e2c4c11197cd4b9e66d10192b234e51f864\": rpc error: code = NotFound desc = could not find container \"97cb4f814576543ee642a0a0a5995e2c4c11197cd4b9e66d10192b234e51f864\": container with ID starting with 97cb4f814576543ee642a0a0a5995e2c4c11197cd4b9e66d10192b234e51f864 not found: ID does not exist"
Oct 31 01:49:20.527593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.527478    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"]
Oct 31 01:49:20.531578 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:20.531545    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cb5756cb-openshift-default-9785d8d5c-598zd"]
Oct 31 01:49:21.675776 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:21.675743    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl"]
Oct 31 01:49:21.676417 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:21.676403    5650 cpu_manager.go:414] "containerMap: removing container" podUID="1357b02f-b722-40ac-8ec2-35f5626d8347" containerName="istio-proxy"
Oct 31 01:49:21.676496 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:21.676418    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1357b02f-b722-40ac-8ec2-35f5626d8347" containerName="istio-proxy"
Oct 31 01:49:21.676496 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:21.676430    5650 cpu_manager.go:414] "containerMap: removing container" podUID="1357b02f-b722-40ac-8ec2-35f5626d8347" containerName="istio-proxy"
Oct 31 01:49:21.676496 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:21.676435    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1357b02f-b722-40ac-8ec2-35f5626d8347" containerName="istio-proxy"
Oct 31 01:49:21.676590 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:21.676578    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="1357b02f-b722-40ac-8ec2-35f5626d8347" containerName="istio-proxy"
Oct 31 01:49:21.676627 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:21.676590    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="1357b02f-b722-40ac-8ec2-35f5626d8347" containerName="istio-proxy"
Oct 31 01:49:21.681558 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:21.681532    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl"
Oct 31 01:49:21.689924 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:21.689898    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl"]
Oct 31 01:49:21.690440 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:21.690420    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-676801-dockercfg-gz85k\""
Oct 31 01:49:21.690577 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:21.690429    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-676801\""
Oct 31 01:49:21.739657 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:21.739626    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-676801-84b8cddcf-fm7th"]
Oct 31 01:49:21.744665 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:21.744625    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-676801-84b8cddcf-fm7th"
Oct 31 01:49:21.746148 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:21.746115    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-676801-84b8cddcf-fm7th"]
Oct 31 01:49:21.828469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:21.825715    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chdbd\" (UniqueName: \"kubernetes.io/projected/2708e1b7-59fe-45a0-8b4e-320e8a365b17-kube-api-access-chdbd\") pod \"payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl\" (UID: \"2708e1b7-59fe-45a0-8b4e-320e8a365b17\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl"
Oct 31 01:49:21.828469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:21.825961    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2708e1b7-59fe-45a0-8b4e-320e8a365b17-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl\" (UID: \"2708e1b7-59fe-45a0-8b4e-320e8a365b17\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl"
Oct 31 01:49:21.939526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:21.939098    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chdbd\" (UniqueName: \"kubernetes.io/projected/2708e1b7-59fe-45a0-8b4e-320e8a365b17-kube-api-access-chdbd\") pod \"payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl\" (UID: \"2708e1b7-59fe-45a0-8b4e-320e8a365b17\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl"
Oct 31 01:49:21.939526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:21.939178    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxdc8\" (UniqueName: \"kubernetes.io/projected/7b03b560-9699-4e2c-8970-5e7cf56c1cf9-kube-api-access-dxdc8\") pod \"payload-processing-e2e-api-a-676801-84b8cddcf-fm7th\" (UID: \"7b03b560-9699-4e2c-8970-5e7cf56c1cf9\") " pod="openshift-ingress/payload-processing-e2e-api-a-676801-84b8cddcf-fm7th"
Oct 31 01:49:21.939526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:21.939337    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2708e1b7-59fe-45a0-8b4e-320e8a365b17-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl\" (UID: \"2708e1b7-59fe-45a0-8b4e-320e8a365b17\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl"
Oct 31 01:49:21.939526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:21.939378    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b03b560-9699-4e2c-8970-5e7cf56c1cf9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-676801-84b8cddcf-fm7th\" (UID: \"7b03b560-9699-4e2c-8970-5e7cf56c1cf9\") " pod="openshift-ingress/payload-processing-e2e-api-a-676801-84b8cddcf-fm7th"
Oct 31 01:49:21.952469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:21.940258    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2708e1b7-59fe-45a0-8b4e-320e8a365b17-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl\" (UID: \"2708e1b7-59fe-45a0-8b4e-320e8a365b17\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl"
Oct 31 01:49:21.974533 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:21.967760    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chdbd\" (UniqueName: \"kubernetes.io/projected/2708e1b7-59fe-45a0-8b4e-320e8a365b17-kube-api-access-chdbd\") pod \"payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl\" (UID: \"2708e1b7-59fe-45a0-8b4e-320e8a365b17\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl"
Oct 31 01:49:22.000773 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:22.000735    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl"
Oct 31 01:49:22.049464 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:22.049257    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxdc8\" (UniqueName: \"kubernetes.io/projected/7b03b560-9699-4e2c-8970-5e7cf56c1cf9-kube-api-access-dxdc8\") pod \"payload-processing-e2e-api-a-676801-84b8cddcf-fm7th\" (UID: \"7b03b560-9699-4e2c-8970-5e7cf56c1cf9\") " pod="openshift-ingress/payload-processing-e2e-api-a-676801-84b8cddcf-fm7th"
Oct 31 01:49:22.049665 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:22.049477    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b03b560-9699-4e2c-8970-5e7cf56c1cf9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-676801-84b8cddcf-fm7th\" (UID: \"7b03b560-9699-4e2c-8970-5e7cf56c1cf9\") " pod="openshift-ingress/payload-processing-e2e-api-a-676801-84b8cddcf-fm7th"
Oct 31 01:49:22.050525 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:22.050497    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b03b560-9699-4e2c-8970-5e7cf56c1cf9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-676801-84b8cddcf-fm7th\" (UID: \"7b03b560-9699-4e2c-8970-5e7cf56c1cf9\") " pod="openshift-ingress/payload-processing-e2e-api-a-676801-84b8cddcf-fm7th"
Oct 31 01:49:22.070828 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:22.070796    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxdc8\" (UniqueName: \"kubernetes.io/projected/7b03b560-9699-4e2c-8970-5e7cf56c1cf9-kube-api-access-dxdc8\") pod \"payload-processing-e2e-api-a-676801-84b8cddcf-fm7th\" (UID: \"7b03b560-9699-4e2c-8970-5e7cf56c1cf9\") " pod="openshift-ingress/payload-processing-e2e-api-a-676801-84b8cddcf-fm7th"
Oct 31 01:49:22.152412 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:22.152379    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1357b02f-b722-40ac-8ec2-35f5626d8347" path="/var/lib/kubelet/pods/1357b02f-b722-40ac-8ec2-35f5626d8347/volumes"
Oct 31 01:49:22.363382 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:22.363353    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-676801-84b8cddcf-fm7th"
Oct 31 01:49:22.380432 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:22.380402    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl"]
Oct 31 01:49:22.697178 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:49:22.697139    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b03b560_9699_4e2c_8970_5e7cf56c1cf9.slice/crio-94da82bc1c4872af82f3e3d99d7969f42e6fa7c74c07d8f3c0215cd075a00c8f WatchSource:0}: Error finding container 94da82bc1c4872af82f3e3d99d7969f42e6fa7c74c07d8f3c0215cd075a00c8f: Status 404 returned error can't find the container with id 94da82bc1c4872af82f3e3d99d7969f42e6fa7c74c07d8f3c0215cd075a00c8f
Oct 31 01:49:22.697815 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:22.697743    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-676801-84b8cddcf-fm7th"]
Oct 31 01:49:23.243150 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:23.243112    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-676801-84b8cddcf-fm7th" event={"ID":"7b03b560-9699-4e2c-8970-5e7cf56c1cf9","Type":"ContainerStarted","Data":"e1d456aa58de01741bdae565c7710d7870821541664bdadda43a57a8c424ebf4"}
Oct 31 01:49:23.243416 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:23.243396    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-676801-84b8cddcf-fm7th" event={"ID":"7b03b560-9699-4e2c-8970-5e7cf56c1cf9","Type":"ContainerStarted","Data":"94da82bc1c4872af82f3e3d99d7969f42e6fa7c74c07d8f3c0215cd075a00c8f"}
Oct 31 01:49:23.249008 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:23.246662    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-676801-84b8cddcf-fm7th"
Oct 31 01:49:23.253948 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:23.253908    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl" event={"ID":"2708e1b7-59fe-45a0-8b4e-320e8a365b17","Type":"ContainerStarted","Data":"1e595dfcc3b5e9d3bf46940ed82182979ee20830372e2c9d22b8693f01038e5d"}
Oct 31 01:49:23.254542 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:23.254139    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl"
Oct 31 01:49:23.254542 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:23.254186    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl" event={"ID":"2708e1b7-59fe-45a0-8b4e-320e8a365b17","Type":"ContainerStarted","Data":"75ae03ac881d43ce4798dcea625c0e13242b7f359e729b3645a38e2a2d09ad12"}
Oct 31 01:49:23.273234 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:23.273161    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-676801-84b8cddcf-fm7th" podStartSLOduration=2.273145485 podStartE2EDuration="2.273145485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:23.270404234 +0000 UTC m=+1719.274118532" watchObservedRunningTime="2026-10-31 01:49:23.273145485 +0000 UTC m=+1719.276859805"
Oct 31 01:49:23.295943 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:23.295878    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl" podStartSLOduration=2.295857575 podStartE2EDuration="2.295857575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:23.293835899 +0000 UTC m=+1719.297550190" watchObservedRunningTime="2026-10-31 01:49:23.295857575 +0000 UTC m=+1719.299571868"
Oct 31 01:49:25.112496 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:25.112436    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g"
Oct 31 01:49:25.112948 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:25.112535    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m"
Oct 31 01:49:26.028390 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:26.028356    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e3532af-4nsks"]
Oct 31 01:49:26.034893 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:26.034862    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e3532af-4nsks"]
Oct 31 01:49:26.122344 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:26.122302    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a681024-af72-4506-a3ca-97e20cfe4e1c" path="/var/lib/kubelet/pods/3a681024-af72-4506-a3ca-97e20cfe4e1c/volumes"
Oct 31 01:49:28.556834 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:49:28.556796    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:49:29.310619 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:29.310581    5650 generic.go:356] "Generic (PLEG): container finished" podID="868f691d-84c7-4301-b42f-3a314ffcdb0b" containerID="7d2c5aa798f3989219b918d5ba62db13055c9cb10d601bba4a8940b89640e904" exitCode=0
Oct 31 01:49:29.310838 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:29.310655    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2" event={"ID":"868f691d-84c7-4301-b42f-3a314ffcdb0b","Type":"ContainerDied","Data":"7d2c5aa798f3989219b918d5ba62db13055c9cb10d601bba4a8940b89640e904"}
Oct 31 01:49:29.311212 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:29.311195    5650 scope.go:122] "RemoveContainer" containerID="7d2c5aa798f3989219b918d5ba62db13055c9cb10d601bba4a8940b89640e904"
Oct 31 01:49:29.312945 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:29.312911    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:49:29.313038 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:29.312986    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:49:29.313038 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:29.313018    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:49:30.336110 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:30.336072    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2" event={"ID":"868f691d-84c7-4301-b42f-3a314ffcdb0b","Type":"ContainerStarted","Data":"b0425641d7c3e62116c31d4cf3feca2ef23c72081857bdb6c75170da706a43d0"}
Oct 31 01:49:31.272758 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:31.272724    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:49:31.274015 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:31.273982    5650 patch_prober.go:29] interesting pod/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 31 01:49:31.274164 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:31.274055    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2" podUID="868f691d-84c7-4301-b42f-3a314ffcdb0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 31 01:49:31.275844 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:31.275820    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g"]
Oct 31 01:49:31.336514 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:31.336477    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m"]
Oct 31 01:49:32.272690 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:32.272650    5650 patch_prober.go:29] interesting pod/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 31 01:49:32.272872 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:32.272724    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2" podUID="868f691d-84c7-4301-b42f-3a314ffcdb0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 31 01:49:33.273018 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:33.272984    5650 patch_prober.go:29] interesting pod/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 31 01:49:33.273422 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:33.273045    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2" podUID="868f691d-84c7-4301-b42f-3a314ffcdb0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 31 01:49:34.267058 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:34.267029    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-676801-84b8cddcf-fm7th"
Oct 31 01:49:34.267248 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:34.267103    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl"
Oct 31 01:49:34.272415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:34.272377    5650 patch_prober.go:29] interesting pod/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 31 01:49:34.272589 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:34.272433    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2" podUID="868f691d-84c7-4301-b42f-3a314ffcdb0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 31 01:49:35.272943 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:35.272903    5650 patch_prober.go:29] interesting pod/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 31 01:49:35.273338 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:35.272963    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2" podUID="868f691d-84c7-4301-b42f-3a314ffcdb0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 31 01:49:35.705537 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:35.705496    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:49:36.277052 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:36.277011    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g" podUID="06974ced-b6e7-4dc3-9003-1a1d6d8acd12" containerName="payload-processing" containerID="cri-o://3b3d30a59149f06412d7a5a6884f664632444e7a6fa337778aaaeb0de44edd0e" gracePeriod=25
Oct 31 01:49:36.277822 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:36.277603    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:49:36.337750 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:36.337702    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m" podUID="1571b2ff-b45f-44ae-bc9c-4043fd61a2b0" containerName="payload-pre-processing" containerID="cri-o://bc73861d6e2443dd9628f546be4b87a7907ae2bcfd7c21082b31c32b895b76e9" gracePeriod=25
Oct 31 01:49:36.419405 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:36.419372    5650 generic.go:356] "Generic (PLEG): container finished" podID="06974ced-b6e7-4dc3-9003-1a1d6d8acd12" containerID="3b3d30a59149f06412d7a5a6884f664632444e7a6fa337778aaaeb0de44edd0e" exitCode=0
Oct 31 01:49:36.419593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:36.419455    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g" event={"ID":"06974ced-b6e7-4dc3-9003-1a1d6d8acd12","Type":"ContainerDied","Data":"3b3d30a59149f06412d7a5a6884f664632444e7a6fa337778aaaeb0de44edd0e"}
Oct 31 01:49:36.420695 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:36.420675    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:49:36.696412 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:36.694673    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g"
Oct 31 01:49:36.730732 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:36.730699    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m"
Oct 31 01:49:36.766916 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:36.766884    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06974ced-b6e7-4dc3-9003-1a1d6d8acd12-kube-api-access-fvw2r\" (UniqueName: \"kubernetes.io/projected/06974ced-b6e7-4dc3-9003-1a1d6d8acd12-kube-api-access-fvw2r\") pod \"06974ced-b6e7-4dc3-9003-1a1d6d8acd12\" (UID: \"06974ced-b6e7-4dc3-9003-1a1d6d8acd12\") "
Oct 31 01:49:36.767097 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:36.767044    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1571b2ff-b45f-44ae-bc9c-4043fd61a2b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1571b2ff-b45f-44ae-bc9c-4043fd61a2b0-plugins-config-volume\") pod \"1571b2ff-b45f-44ae-bc9c-4043fd61a2b0\" (UID: \"1571b2ff-b45f-44ae-bc9c-4043fd61a2b0\") "
Oct 31 01:49:36.767230 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:36.767216    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1571b2ff-b45f-44ae-bc9c-4043fd61a2b0-kube-api-access-6ssxk\" (UniqueName: \"kubernetes.io/projected/1571b2ff-b45f-44ae-bc9c-4043fd61a2b0-kube-api-access-6ssxk\") pod \"1571b2ff-b45f-44ae-bc9c-4043fd61a2b0\" (UID: \"1571b2ff-b45f-44ae-bc9c-4043fd61a2b0\") "
Oct 31 01:49:36.767379 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:36.767366    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06974ced-b6e7-4dc3-9003-1a1d6d8acd12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06974ced-b6e7-4dc3-9003-1a1d6d8acd12-plugins-config-volume\") pod \"06974ced-b6e7-4dc3-9003-1a1d6d8acd12\" (UID: \"06974ced-b6e7-4dc3-9003-1a1d6d8acd12\") "
Oct 31 01:49:36.767560 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:36.767532    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1571b2ff-b45f-44ae-bc9c-4043fd61a2b0-plugins-config-volume" pod "1571b2ff-b45f-44ae-bc9c-4043fd61a2b0" (UID: "1571b2ff-b45f-44ae-bc9c-4043fd61a2b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:49:36.767848 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:36.767828    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06974ced-b6e7-4dc3-9003-1a1d6d8acd12-plugins-config-volume" pod "06974ced-b6e7-4dc3-9003-1a1d6d8acd12" (UID: "06974ced-b6e7-4dc3-9003-1a1d6d8acd12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:49:36.768145 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:36.768119    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06974ced-b6e7-4dc3-9003-1a1d6d8acd12-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:36.768228 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:36.768151    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1571b2ff-b45f-44ae-bc9c-4043fd61a2b0-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:36.771098 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:36.771073    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1571b2ff-b45f-44ae-bc9c-4043fd61a2b0-kube-api-access-6ssxk" pod "1571b2ff-b45f-44ae-bc9c-4043fd61a2b0" (UID: "1571b2ff-b45f-44ae-bc9c-4043fd61a2b0"). InnerVolumeSpecName "kube-api-access-6ssxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:49:36.786291 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:36.786258    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06974ced-b6e7-4dc3-9003-1a1d6d8acd12-kube-api-access-fvw2r" pod "06974ced-b6e7-4dc3-9003-1a1d6d8acd12" (UID: "06974ced-b6e7-4dc3-9003-1a1d6d8acd12"). InnerVolumeSpecName "kube-api-access-fvw2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:49:36.869733 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:36.869638    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvw2r\" (UniqueName: \"kubernetes.io/projected/06974ced-b6e7-4dc3-9003-1a1d6d8acd12-kube-api-access-fvw2r\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:36.869962 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:36.869930    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ssxk\" (UniqueName: \"kubernetes.io/projected/1571b2ff-b45f-44ae-bc9c-4043fd61a2b0-kube-api-access-6ssxk\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:49:37.430264 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:37.430232    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g"
Oct 31 01:49:37.430777 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:37.430227    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g" event={"ID":"06974ced-b6e7-4dc3-9003-1a1d6d8acd12","Type":"ContainerDied","Data":"4e90228c81e4bb44a9b9442b97bb36d2070efd5a6ae381a633fbe7f9411fd281"}
Oct 31 01:49:37.430777 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:37.430371    5650 scope.go:122] "RemoveContainer" containerID="3b3d30a59149f06412d7a5a6884f664632444e7a6fa337778aaaeb0de44edd0e"
Oct 31 01:49:37.432667 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:37.432638    5650 generic.go:356] "Generic (PLEG): container finished" podID="1571b2ff-b45f-44ae-bc9c-4043fd61a2b0" containerID="bc73861d6e2443dd9628f546be4b87a7907ae2bcfd7c21082b31c32b895b76e9" exitCode=0
Oct 31 01:49:37.432825 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:37.432763    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m" event={"ID":"1571b2ff-b45f-44ae-bc9c-4043fd61a2b0","Type":"ContainerDied","Data":"bc73861d6e2443dd9628f546be4b87a7907ae2bcfd7c21082b31c32b895b76e9"}
Oct 31 01:49:37.432825 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:37.432821    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m" event={"ID":"1571b2ff-b45f-44ae-bc9c-4043fd61a2b0","Type":"ContainerDied","Data":"0ae5b81a4e4f7deadc66e4a634c8251feedd174f5ed291236a3aa4897e9a607f"}
Oct 31 01:49:37.432947 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:37.432834    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m"
Oct 31 01:49:37.454272 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:37.454243    5650 scope.go:122] "RemoveContainer" containerID="bc73861d6e2443dd9628f546be4b87a7907ae2bcfd7c21082b31c32b895b76e9"
Oct 31 01:49:37.466297 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:37.466262    5650 scope.go:122] "RemoveContainer" containerID="bc73861d6e2443dd9628f546be4b87a7907ae2bcfd7c21082b31c32b895b76e9"
Oct 31 01:49:37.466730 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:49:37.466702    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc73861d6e2443dd9628f546be4b87a7907ae2bcfd7c21082b31c32b895b76e9\": container with ID starting with bc73861d6e2443dd9628f546be4b87a7907ae2bcfd7c21082b31c32b895b76e9 not found: ID does not exist" containerID="bc73861d6e2443dd9628f546be4b87a7907ae2bcfd7c21082b31c32b895b76e9"
Oct 31 01:49:37.466854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:37.466734    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc73861d6e2443dd9628f546be4b87a7907ae2bcfd7c21082b31c32b895b76e9"} err="failed to get container status \"bc73861d6e2443dd9628f546be4b87a7907ae2bcfd7c21082b31c32b895b76e9\": rpc error: code = NotFound desc = could not find container \"bc73861d6e2443dd9628f546be4b87a7907ae2bcfd7c21082b31c32b895b76e9\": container with ID starting with bc73861d6e2443dd9628f546be4b87a7907ae2bcfd7c21082b31c32b895b76e9 not found: ID does not exist"
Oct 31 01:49:37.467502 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:37.467436    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m"]
Oct 31 01:49:37.472050 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:37.472027    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-d6d5c769f-jff2m"]
Oct 31 01:49:37.484834 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:37.484808    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g"]
Oct 31 01:49:37.489297 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:37.489269    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-073e24-84448c64c-2cs2g"]
Oct 31 01:49:38.120053 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:38.120017    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06974ced-b6e7-4dc3-9003-1a1d6d8acd12" path="/var/lib/kubelet/pods/06974ced-b6e7-4dc3-9003-1a1d6d8acd12/volumes"
Oct 31 01:49:38.120546 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:38.120531    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1571b2ff-b45f-44ae-bc9c-4043fd61a2b0" path="/var/lib/kubelet/pods/1571b2ff-b45f-44ae-bc9c-4043fd61a2b0/volumes"
Oct 31 01:49:39.852113 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:39.852076    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl"]
Oct 31 01:49:39.852737 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:39.852722    5650 cpu_manager.go:414] "containerMap: removing container" podUID="1571b2ff-b45f-44ae-bc9c-4043fd61a2b0" containerName="payload-pre-processing"
Oct 31 01:49:39.852794 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:39.852738    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1571b2ff-b45f-44ae-bc9c-4043fd61a2b0" containerName="payload-pre-processing"
Oct 31 01:49:39.852794 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:39.852765    5650 cpu_manager.go:414] "containerMap: removing container" podUID="06974ced-b6e7-4dc3-9003-1a1d6d8acd12" containerName="payload-processing"
Oct 31 01:49:39.852794 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:39.852770    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06974ced-b6e7-4dc3-9003-1a1d6d8acd12" containerName="payload-processing"
Oct 31 01:49:39.852885 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:39.852865    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="1571b2ff-b45f-44ae-bc9c-4043fd61a2b0" containerName="payload-pre-processing"
Oct 31 01:49:39.852885 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:39.852877    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="06974ced-b6e7-4dc3-9003-1a1d6d8acd12" containerName="payload-processing"
Oct 31 01:49:39.858068 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:39.858041    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl"
Oct 31 01:49:39.864070 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:39.863953    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-073e24\""
Oct 31 01:49:39.864070 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:39.863972    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-073e24-dockercfg-6zzn8\""
Oct 31 01:49:39.867171 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:39.866956    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl"]
Oct 31 01:49:39.919853 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:39.919813    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744d9078-58b7-4c9c-b92d-e36ff748e020-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl\" (UID: \"744d9078-58b7-4c9c-b92d-e36ff748e020\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl"
Oct 31 01:49:39.920133 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:39.919913    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9zz9\" (UniqueName: \"kubernetes.io/projected/744d9078-58b7-4c9c-b92d-e36ff748e020-kube-api-access-f9zz9\") pod \"payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl\" (UID: \"744d9078-58b7-4c9c-b92d-e36ff748e020\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl"
Oct 31 01:49:40.027525 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:40.027482    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744d9078-58b7-4c9c-b92d-e36ff748e020-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl\" (UID: \"744d9078-58b7-4c9c-b92d-e36ff748e020\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl"
Oct 31 01:49:40.027741 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:40.027670    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9zz9\" (UniqueName: \"kubernetes.io/projected/744d9078-58b7-4c9c-b92d-e36ff748e020-kube-api-access-f9zz9\") pod \"payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl\" (UID: \"744d9078-58b7-4c9c-b92d-e36ff748e020\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl"
Oct 31 01:49:40.028779 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:40.028754    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744d9078-58b7-4c9c-b92d-e36ff748e020-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl\" (UID: \"744d9078-58b7-4c9c-b92d-e36ff748e020\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl"
Oct 31 01:49:40.066009 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:40.065966    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9zz9\" (UniqueName: \"kubernetes.io/projected/744d9078-58b7-4c9c-b92d-e36ff748e020-kube-api-access-f9zz9\") pod \"payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl\" (UID: \"744d9078-58b7-4c9c-b92d-e36ff748e020\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl"
Oct 31 01:49:40.175079 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:40.174983    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl"
Oct 31 01:49:40.497091 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:40.497051    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl"]
Oct 31 01:49:40.499519 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:49:40.499482    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744d9078_58b7_4c9c_b92d_e36ff748e020.slice/crio-9c342d69dc06b4b809ab4b705292c3bc3bcd60ee001f57d952c47bec1bc44ed2 WatchSource:0}: Error finding container 9c342d69dc06b4b809ab4b705292c3bc3bcd60ee001f57d952c47bec1bc44ed2: Status 404 returned error can't find the container with id 9c342d69dc06b4b809ab4b705292c3bc3bcd60ee001f57d952c47bec1bc44ed2
Oct 31 01:49:41.404723 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.404692    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"]
Oct 31 01:49:41.409133 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.409094    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.419886 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.419856    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e52ba1-openshift-default-dockercfg-xrxmn\""
Oct 31 01:49:41.432514 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.432478    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"]
Oct 31 01:49:41.452491 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.452431    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-credential-socket\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.452731 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.452512    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-envoy\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.452731 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.452613    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s54xb\" (UniqueName: \"kubernetes.io/projected/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-kube-api-access-s54xb\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.452731 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.452646    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istiod-ca-cert\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.452974 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.452762    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-workload-certs\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.452974 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.452801    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-podinfo\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.452974 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.452894    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-workload-socket\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.452974 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.452955    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-data\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.453303 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.453017    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-token\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.479056 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.479013    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl" event={"ID":"744d9078-58b7-4c9c-b92d-e36ff748e020","Type":"ContainerStarted","Data":"9c342d69dc06b4b809ab4b705292c3bc3bcd60ee001f57d952c47bec1bc44ed2"}
Oct 31 01:49:41.554897 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.554802    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-token\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.555083 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.555045    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-credential-socket\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.555168 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.555086    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-envoy\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.555231 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.555173    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s54xb\" (UniqueName: \"kubernetes.io/projected/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-kube-api-access-s54xb\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.555231 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.555213    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istiod-ca-cert\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.555339 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.555324    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-workload-certs\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.555397 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.555365    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-podinfo\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.555501 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.555477    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-workload-socket\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.555559 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.555544    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-data\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.556021 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.555992    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-data\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.556639 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.556606    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-credential-socket\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.557331 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.557308    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-workload-certs\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.557752 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.557726    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-workload-socket\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.557907 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.557885    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istiod-ca-cert\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.560094 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.560069    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-envoy\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.561175 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.561150    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-podinfo\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.569239 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.569206    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-token\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.569793 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.569771    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s54xb\" (UniqueName: \"kubernetes.io/projected/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-kube-api-access-s54xb\") pod \"e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") " pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:41.738869 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:41.737650    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:42.115635 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:49:42.115601    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f59db2_d04e_4e96_bf12_5a7d3f18eb99.slice/crio-7edf7de22c15edb070e4961486d1fc7e4f1082274b518b5de6ea351eca861e1b WatchSource:0}: Error finding container 7edf7de22c15edb070e4961486d1fc7e4f1082274b518b5de6ea351eca861e1b: Status 404 returned error can't find the container with id 7edf7de22c15edb070e4961486d1fc7e4f1082274b518b5de6ea351eca861e1b
Oct 31 01:49:42.122767 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:42.122724    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:49:42.122879 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:42.122814    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:49:42.122879 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:42.122856    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:49:42.132386 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:42.132289    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"]
Oct 31 01:49:42.491564 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:42.491519    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td" event={"ID":"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99","Type":"ContainerStarted","Data":"2f447df343bf3d68976ec259c95615222060b10afd976e1cefaabe5d3ddcf293"}
Oct 31 01:49:42.492094 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:42.491629    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td" event={"ID":"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99","Type":"ContainerStarted","Data":"7edf7de22c15edb070e4961486d1fc7e4f1082274b518b5de6ea351eca861e1b"}
Oct 31 01:49:42.524920 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:42.524848    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td" podStartSLOduration=1.524828259 podStartE2EDuration="1.524828259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:42.519693367 +0000 UTC m=+1738.523407658" watchObservedRunningTime="2026-10-31 01:49:42.524828259 +0000 UTC m=+1738.528542551"
Oct 31 01:49:42.738915 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:42.738864    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:43.507745 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:43.507703    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl" event={"ID":"744d9078-58b7-4c9c-b92d-e36ff748e020","Type":"ContainerStarted","Data":"ff7dcec315e51adb18798e79342c018b216f75026cff0694ec4463c304f82acf"}
Oct 31 01:49:43.508565 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:43.508533    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl"
Oct 31 01:49:43.755468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:43.755297    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:43.810941 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:43.810823    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl" podStartSLOduration=2.444479097 podStartE2EDuration="4.810807363s" totalImagesPullingTime="2.366328266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:43.533723461 +0000 UTC m=+1739.537437748" watchObservedRunningTime="2026-10-31 01:49:43.810807363 +0000 UTC m=+1739.814521674"
Oct 31 01:49:44.028962 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:44.028922    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49e8ad58-2dspg"]
Oct 31 01:49:44.043569 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:44.043536    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49e8ad58-2dspg"]
Oct 31 01:49:44.126571 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:44.123432    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ecc2be1-1e13-4564-b18d-8256a69cf828" path="/var/lib/kubelet/pods/3ecc2be1-1e13-4564-b18d-8256a69cf828/volumes"
Oct 31 01:49:44.530555 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:44.529647    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:44.531045 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:44.530983    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:49:46.230303 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:46.230269    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl"]
Oct 31 01:49:46.235723 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:46.235703    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl"
Oct 31 01:49:46.241217 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:46.241187    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e52ba1\""
Oct 31 01:49:46.244783 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:46.244755    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl"]
Oct 31 01:49:46.266333 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:46.266294    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww"]
Oct 31 01:49:46.270744 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:46.270720    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww"
Oct 31 01:49:46.285813 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:46.285783    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e52ba1-dockercfg-rl4cs\""
Oct 31 01:49:46.293669 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:46.293638    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww"]
Oct 31 01:49:46.374516 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:46.368293    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdhxr\" (UniqueName: \"kubernetes.io/projected/9cce0ae0-583a-4f7f-af47-0b5b8fde042b-kube-api-access-vdhxr\") pod \"payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl\" (UID: \"9cce0ae0-583a-4f7f-af47-0b5b8fde042b\") " pod="openshift-ingress/payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl"
Oct 31 01:49:46.374516 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:46.368362    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft4r6\" (UniqueName: \"kubernetes.io/projected/35182dd9-f44c-4d37-b9de-979018244405-kube-api-access-ft4r6\") pod \"payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww\" (UID: \"35182dd9-f44c-4d37-b9de-979018244405\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww"
Oct 31 01:49:46.374516 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:46.368556    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35182dd9-f44c-4d37-b9de-979018244405-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww\" (UID: \"35182dd9-f44c-4d37-b9de-979018244405\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww"
Oct 31 01:49:46.374516 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:46.368781    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cce0ae0-583a-4f7f-af47-0b5b8fde042b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl\" (UID: \"9cce0ae0-583a-4f7f-af47-0b5b8fde042b\") " pod="openshift-ingress/payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl"
Oct 31 01:49:46.476467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:46.471891    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cce0ae0-583a-4f7f-af47-0b5b8fde042b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl\" (UID: \"9cce0ae0-583a-4f7f-af47-0b5b8fde042b\") " pod="openshift-ingress/payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl"
Oct 31 01:49:46.476467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:46.472152    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdhxr\" (UniqueName: \"kubernetes.io/projected/9cce0ae0-583a-4f7f-af47-0b5b8fde042b-kube-api-access-vdhxr\") pod \"payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl\" (UID: \"9cce0ae0-583a-4f7f-af47-0b5b8fde042b\") " pod="openshift-ingress/payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl"
Oct 31 01:49:46.476467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:46.472226    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft4r6\" (UniqueName: \"kubernetes.io/projected/35182dd9-f44c-4d37-b9de-979018244405-kube-api-access-ft4r6\") pod \"payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww\" (UID: \"35182dd9-f44c-4d37-b9de-979018244405\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww"
Oct 31 01:49:46.476467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:46.474290    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35182dd9-f44c-4d37-b9de-979018244405-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww\" (UID: \"35182dd9-f44c-4d37-b9de-979018244405\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww"
Oct 31 01:49:46.476467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:46.475519    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35182dd9-f44c-4d37-b9de-979018244405-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww\" (UID: \"35182dd9-f44c-4d37-b9de-979018244405\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww"
Oct 31 01:49:46.491592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:46.482700    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cce0ae0-583a-4f7f-af47-0b5b8fde042b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl\" (UID: \"9cce0ae0-583a-4f7f-af47-0b5b8fde042b\") " pod="openshift-ingress/payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl"
Oct 31 01:49:46.507966 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:46.507205    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdhxr\" (UniqueName: \"kubernetes.io/projected/9cce0ae0-583a-4f7f-af47-0b5b8fde042b-kube-api-access-vdhxr\") pod \"payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl\" (UID: \"9cce0ae0-583a-4f7f-af47-0b5b8fde042b\") " pod="openshift-ingress/payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl"
Oct 31 01:49:46.513171 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:46.513134    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft4r6\" (UniqueName: \"kubernetes.io/projected/35182dd9-f44c-4d37-b9de-979018244405-kube-api-access-ft4r6\") pod \"payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww\" (UID: \"35182dd9-f44c-4d37-b9de-979018244405\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww"
Oct 31 01:49:46.557047 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:46.557011    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl"
Oct 31 01:49:46.596971 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:46.596940    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww"
Oct 31 01:49:46.846870 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:49:46.846836    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cce0ae0_583a_4f7f_af47_0b5b8fde042b.slice/crio-61ace603c07dd13971257ed005e962a3fa5cb97e7e7ae14f69819a45aede4605 WatchSource:0}: Error finding container 61ace603c07dd13971257ed005e962a3fa5cb97e7e7ae14f69819a45aede4605: Status 404 returned error can't find the container with id 61ace603c07dd13971257ed005e962a3fa5cb97e7e7ae14f69819a45aede4605
Oct 31 01:49:46.847974 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:46.847938    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl"]
Oct 31 01:49:46.911734 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:49:46.911705    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35182dd9_f44c_4d37_b9de_979018244405.slice/crio-d8e4663c7d84b667fafebde4a6469ec14c24b0fca66b9d32aa03f4b52853267b WatchSource:0}: Error finding container d8e4663c7d84b667fafebde4a6469ec14c24b0fca66b9d32aa03f4b52853267b: Status 404 returned error can't find the container with id d8e4663c7d84b667fafebde4a6469ec14c24b0fca66b9d32aa03f4b52853267b
Oct 31 01:49:46.912123 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:46.912100    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww"]
Oct 31 01:49:47.555006 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:47.554968    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl" event={"ID":"9cce0ae0-583a-4f7f-af47-0b5b8fde042b","Type":"ContainerStarted","Data":"b20a32d2e3ba1e811ca3980dd61da593c80d447a59be8844f4acf2bd7e9c0021"}
Oct 31 01:49:47.555506 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:47.555054    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl" event={"ID":"9cce0ae0-583a-4f7f-af47-0b5b8fde042b","Type":"ContainerStarted","Data":"61ace603c07dd13971257ed005e962a3fa5cb97e7e7ae14f69819a45aede4605"}
Oct 31 01:49:47.555506 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:47.555082    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl"
Oct 31 01:49:47.556860 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:47.556835    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww" event={"ID":"35182dd9-f44c-4d37-b9de-979018244405","Type":"ContainerStarted","Data":"247c273cc407bdc345fd1aef4343cc4b983600172781d18a7d554e2ca8beba1b"}
Oct 31 01:49:47.556965 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:47.556883    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww" event={"ID":"35182dd9-f44c-4d37-b9de-979018244405","Type":"ContainerStarted","Data":"d8e4663c7d84b667fafebde4a6469ec14c24b0fca66b9d32aa03f4b52853267b"}
Oct 31 01:49:47.556965 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:47.556937    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww"
Oct 31 01:49:47.578753 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:47.578695    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl" podStartSLOduration=1.578679448 podStartE2EDuration="1.578679448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:47.57574362 +0000 UTC m=+1743.579457912" watchObservedRunningTime="2026-10-31 01:49:47.578679448 +0000 UTC m=+1743.582393738"
Oct 31 01:49:47.606083 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:47.606019    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww" podStartSLOduration=1.606003906 podStartE2EDuration="1.606003906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:47.603900042 +0000 UTC m=+1743.607614343" watchObservedRunningTime="2026-10-31 01:49:47.606003906 +0000 UTC m=+1743.609718205"
Oct 31 01:49:55.539639 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:55.539600    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl"
Oct 31 01:49:58.554092 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:49:58.554055    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:49:58.567720 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:58.567692    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl"
Oct 31 01:49:58.567900 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:58.567760    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww"
Oct 31 01:49:58.838842 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:58.838748    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"]
Oct 31 01:49:58.843732 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:58.843707    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:58.856138 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:58.851779    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-90c4d1-openshift-default-dockercfg-dzqvs\""
Oct 31 01:49:58.862133 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:58.862100    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"]
Oct 31 01:49:58.897892 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:58.897856    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-workload-certs\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:58.898109 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:58.897901    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-envoy\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:58.898109 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:58.897937    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbe8798-c240-4556-bc46-d28d7ffacef2-istiod-ca-cert\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:58.898109 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:58.898046    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-credential-socket\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:58.898109 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:58.898084    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf2ck\" (UniqueName: \"kubernetes.io/projected/dcbe8798-c240-4556-bc46-d28d7ffacef2-kube-api-access-kf2ck\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:58.898436 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:58.898206    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-workload-socket\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:58.898436 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:58.898292    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-data\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:58.898436 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:58.898347    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-podinfo\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:58.898436 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:58.898370    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-token\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:58.999672 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:58.999634    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-workload-socket\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:58.999887 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:58.999735    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-data\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:58.999887 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:58.999786    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-podinfo\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:58.999887 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:58.999815    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-token\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:59.000057 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:58.999935    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-workload-certs\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:59.000057 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:58.999963    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-envoy\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:59.000057 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:58.999996    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbe8798-c240-4556-bc46-d28d7ffacef2-istiod-ca-cert\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:59.000208 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:59.000123    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-credential-socket\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:59.000208 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:59.000166    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf2ck\" (UniqueName: \"kubernetes.io/projected/dcbe8798-c240-4556-bc46-d28d7ffacef2-kube-api-access-kf2ck\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:59.001623 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:59.001075    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-data\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:59.001623 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:59.001115    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-workload-certs\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:59.001623 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:59.001140    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-workload-socket\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:59.001623 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:59.001488    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-credential-socket\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:59.003510 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:59.001862    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbe8798-c240-4556-bc46-d28d7ffacef2-istiod-ca-cert\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:59.004167 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:59.004139    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-envoy\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:59.006291 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:59.006268    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-podinfo\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:59.015079 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:59.015051    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-token\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:59.015337 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:59.015314    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf2ck\" (UniqueName: \"kubernetes.io/projected/dcbe8798-c240-4556-bc46-d28d7ffacef2-kube-api-access-kf2ck\") pod \"e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") " pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:59.166526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:59.163214    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:49:59.592285 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:59.592252    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"]
Oct 31 01:49:59.592677 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:49:59.592435    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbe8798_c240_4556_bc46_d28d7ffacef2.slice/crio-511d695da407518028b6ff617906a779518e1a4f045d98a7585ed2eb1d707c29 WatchSource:0}: Error finding container 511d695da407518028b6ff617906a779518e1a4f045d98a7585ed2eb1d707c29: Status 404 returned error can't find the container with id 511d695da407518028b6ff617906a779518e1a4f045d98a7585ed2eb1d707c29
Oct 31 01:49:59.595801 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:59.595750    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:49:59.595969 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:59.595845    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:49:59.595969 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:59.595888    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:49:59.683353 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:59.683319    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn" event={"ID":"dcbe8798-c240-4556-bc46-d28d7ffacef2","Type":"ContainerStarted","Data":"87d9887636d1211e30cfd8159074b990b8f029a0c3dc68a70ea34dd8ac4e0c01"}
Oct 31 01:49:59.683534 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:59.683383    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn" event={"ID":"dcbe8798-c240-4556-bc46-d28d7ffacef2","Type":"ContainerStarted","Data":"511d695da407518028b6ff617906a779518e1a4f045d98a7585ed2eb1d707c29"}
Oct 31 01:49:59.733467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:49:59.732464    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn" podStartSLOduration=1.732403062 podStartE2EDuration="1.732403062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:59.730026053 +0000 UTC m=+1755.733740336" watchObservedRunningTime="2026-10-31 01:49:59.732403062 +0000 UTC m=+1755.736117353"
Oct 31 01:50:00.164684 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:00.164646    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:50:00.166135 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:00.166106    5650 patch_prober.go:29] interesting pod/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 31 01:50:00.166243 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:00.166161    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn" podUID="dcbe8798-c240-4556-bc46-d28d7ffacef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 31 01:50:01.026320 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:01.026284    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64bf97a5-t6vfn"]
Oct 31 01:50:01.029947 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:01.029919    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64bf97a5-t6vfn"]
Oct 31 01:50:01.168963 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:01.168933    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:50:01.698536 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:01.698505    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:50:01.699577 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:01.699552    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:50:02.120893 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:02.120859    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="683dbaa0-6bdd-4431-8b3b-6ca550a9f509" path="/var/lib/kubelet/pods/683dbaa0-6bdd-4431-8b3b-6ca550a9f509/volumes"
Oct 31 01:50:13.036756 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:13.036712    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44dafe43-m46n5"]
Oct 31 01:50:13.038641 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:13.038608    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44dafe43-m46n5"]
Oct 31 01:50:14.122686 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:14.122649    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="302ba23c-5ce9-433c-b423-8c4f5ee70d0f" path="/var/lib/kubelet/pods/302ba23c-5ce9-433c-b423-8c4f5ee70d0f/volumes"
Oct 31 01:50:21.535112 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:21.535069    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-676801-84b8cddcf-fm7th"]
Oct 31 01:50:21.542907 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:21.542871    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl"]
Oct 31 01:50:25.803899 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:25.803864    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"]
Oct 31 01:50:25.804631 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:25.804579    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2" podUID="868f691d-84c7-4301-b42f-3a314ffcdb0b" containerName="istio-proxy" containerID="cri-o://b0425641d7c3e62116c31d4cf3feca2ef23c72081857bdb6c75170da706a43d0" gracePeriod=30
Oct 31 01:50:26.542622 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:26.542545    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-676801-84b8cddcf-fm7th" podUID="7b03b560-9699-4e2c-8970-5e7cf56c1cf9" containerName="payload-processing" containerID="cri-o://e1d456aa58de01741bdae565c7710d7870821541664bdadda43a57a8c424ebf4" gracePeriod=25
Oct 31 01:50:26.543610 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:26.543524    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl" podUID="2708e1b7-59fe-45a0-8b4e-320e8a365b17" containerName="payload-pre-processing" containerID="cri-o://1e595dfcc3b5e9d3bf46940ed82182979ee20830372e2c9d22b8693f01038e5d" gracePeriod=25
Oct 31 01:50:26.970776 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:26.970704    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-676801-84b8cddcf-fm7th"
Oct 31 01:50:27.031944 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.031880    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl"
Oct 31 01:50:27.053205 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.051646    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b03b560-9699-4e2c-8970-5e7cf56c1cf9-kube-api-access-dxdc8\" (UniqueName: \"kubernetes.io/projected/7b03b560-9699-4e2c-8970-5e7cf56c1cf9-kube-api-access-dxdc8\") pod \"7b03b560-9699-4e2c-8970-5e7cf56c1cf9\" (UID: \"7b03b560-9699-4e2c-8970-5e7cf56c1cf9\") "
Oct 31 01:50:27.053205 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.051722    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2708e1b7-59fe-45a0-8b4e-320e8a365b17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2708e1b7-59fe-45a0-8b4e-320e8a365b17-plugins-config-volume\") pod \"2708e1b7-59fe-45a0-8b4e-320e8a365b17\" (UID: \"2708e1b7-59fe-45a0-8b4e-320e8a365b17\") "
Oct 31 01:50:27.053205 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.051879    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b03b560-9699-4e2c-8970-5e7cf56c1cf9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b03b560-9699-4e2c-8970-5e7cf56c1cf9-plugins-config-volume\") pod \"7b03b560-9699-4e2c-8970-5e7cf56c1cf9\" (UID: \"7b03b560-9699-4e2c-8970-5e7cf56c1cf9\") "
Oct 31 01:50:27.053205 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.052291    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2708e1b7-59fe-45a0-8b4e-320e8a365b17-kube-api-access-chdbd\" (UniqueName: \"kubernetes.io/projected/2708e1b7-59fe-45a0-8b4e-320e8a365b17-kube-api-access-chdbd\") pod \"2708e1b7-59fe-45a0-8b4e-320e8a365b17\" (UID: \"2708e1b7-59fe-45a0-8b4e-320e8a365b17\") "
Oct 31 01:50:27.053205 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.052315    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2708e1b7-59fe-45a0-8b4e-320e8a365b17-plugins-config-volume" pod "2708e1b7-59fe-45a0-8b4e-320e8a365b17" (UID: "2708e1b7-59fe-45a0-8b4e-320e8a365b17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:27.053205 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.052859    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b03b560-9699-4e2c-8970-5e7cf56c1cf9-plugins-config-volume" pod "7b03b560-9699-4e2c-8970-5e7cf56c1cf9" (UID: "7b03b560-9699-4e2c-8970-5e7cf56c1cf9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:27.056200 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.056135    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2708e1b7-59fe-45a0-8b4e-320e8a365b17-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:27.056200 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.056169    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b03b560-9699-4e2c-8970-5e7cf56c1cf9-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:27.057233 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.057183    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b03b560-9699-4e2c-8970-5e7cf56c1cf9-kube-api-access-dxdc8" pod "7b03b560-9699-4e2c-8970-5e7cf56c1cf9" (UID: "7b03b560-9699-4e2c-8970-5e7cf56c1cf9"). InnerVolumeSpecName "kube-api-access-dxdc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:27.057661 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.057639    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2708e1b7-59fe-45a0-8b4e-320e8a365b17-kube-api-access-chdbd" pod "2708e1b7-59fe-45a0-8b4e-320e8a365b17" (UID: "2708e1b7-59fe-45a0-8b4e-320e8a365b17"). InnerVolumeSpecName "kube-api-access-chdbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:27.072819 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.072777    5650 generic.go:356] "Generic (PLEG): container finished" podID="2708e1b7-59fe-45a0-8b4e-320e8a365b17" containerID="1e595dfcc3b5e9d3bf46940ed82182979ee20830372e2c9d22b8693f01038e5d" exitCode=0
Oct 31 01:50:27.073024 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.072822    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl"
Oct 31 01:50:27.073024 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.072859    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl" event={"ID":"2708e1b7-59fe-45a0-8b4e-320e8a365b17","Type":"ContainerDied","Data":"1e595dfcc3b5e9d3bf46940ed82182979ee20830372e2c9d22b8693f01038e5d"}
Oct 31 01:50:27.073024 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.072925    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl" event={"ID":"2708e1b7-59fe-45a0-8b4e-320e8a365b17","Type":"ContainerDied","Data":"75ae03ac881d43ce4798dcea625c0e13242b7f359e729b3645a38e2a2d09ad12"}
Oct 31 01:50:27.073024 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.072957    5650 scope.go:122] "RemoveContainer" containerID="1e595dfcc3b5e9d3bf46940ed82182979ee20830372e2c9d22b8693f01038e5d"
Oct 31 01:50:27.075142 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.075088    5650 generic.go:356] "Generic (PLEG): container finished" podID="7b03b560-9699-4e2c-8970-5e7cf56c1cf9" containerID="e1d456aa58de01741bdae565c7710d7870821541664bdadda43a57a8c424ebf4" exitCode=0
Oct 31 01:50:27.075142 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.075140    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-676801-84b8cddcf-fm7th"
Oct 31 01:50:27.075297 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.075186    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-676801-84b8cddcf-fm7th" event={"ID":"7b03b560-9699-4e2c-8970-5e7cf56c1cf9","Type":"ContainerDied","Data":"e1d456aa58de01741bdae565c7710d7870821541664bdadda43a57a8c424ebf4"}
Oct 31 01:50:27.075297 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.075253    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-676801-84b8cddcf-fm7th" event={"ID":"7b03b560-9699-4e2c-8970-5e7cf56c1cf9","Type":"ContainerDied","Data":"94da82bc1c4872af82f3e3d99d7969f42e6fa7c74c07d8f3c0215cd075a00c8f"}
Oct 31 01:50:27.101107 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.101076    5650 scope.go:122] "RemoveContainer" containerID="1e595dfcc3b5e9d3bf46940ed82182979ee20830372e2c9d22b8693f01038e5d"
Oct 31 01:50:27.101569 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:50:27.101537    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e595dfcc3b5e9d3bf46940ed82182979ee20830372e2c9d22b8693f01038e5d\": container with ID starting with 1e595dfcc3b5e9d3bf46940ed82182979ee20830372e2c9d22b8693f01038e5d not found: ID does not exist" containerID="1e595dfcc3b5e9d3bf46940ed82182979ee20830372e2c9d22b8693f01038e5d"
Oct 31 01:50:27.101681 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.101621    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e595dfcc3b5e9d3bf46940ed82182979ee20830372e2c9d22b8693f01038e5d"} err="failed to get container status \"1e595dfcc3b5e9d3bf46940ed82182979ee20830372e2c9d22b8693f01038e5d\": rpc error: code = NotFound desc = could not find container \"1e595dfcc3b5e9d3bf46940ed82182979ee20830372e2c9d22b8693f01038e5d\": container with ID starting with 1e595dfcc3b5e9d3bf46940ed82182979ee20830372e2c9d22b8693f01038e5d not found: ID does not exist"
Oct 31 01:50:27.101681 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.101653    5650 scope.go:122] "RemoveContainer" containerID="e1d456aa58de01741bdae565c7710d7870821541664bdadda43a57a8c424ebf4"
Oct 31 01:50:27.113829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.113803    5650 scope.go:122] "RemoveContainer" containerID="e1d456aa58de01741bdae565c7710d7870821541664bdadda43a57a8c424ebf4"
Oct 31 01:50:27.114161 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:50:27.114132    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1d456aa58de01741bdae565c7710d7870821541664bdadda43a57a8c424ebf4\": container with ID starting with e1d456aa58de01741bdae565c7710d7870821541664bdadda43a57a8c424ebf4 not found: ID does not exist" containerID="e1d456aa58de01741bdae565c7710d7870821541664bdadda43a57a8c424ebf4"
Oct 31 01:50:27.114231 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.114173    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1d456aa58de01741bdae565c7710d7870821541664bdadda43a57a8c424ebf4"} err="failed to get container status \"e1d456aa58de01741bdae565c7710d7870821541664bdadda43a57a8c424ebf4\": rpc error: code = NotFound desc = could not find container \"e1d456aa58de01741bdae565c7710d7870821541664bdadda43a57a8c424ebf4\": container with ID starting with e1d456aa58de01741bdae565c7710d7870821541664bdadda43a57a8c424ebf4 not found: ID does not exist"
Oct 31 01:50:27.149333 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.149293    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl"]
Oct 31 01:50:27.154077 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.154046    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-676801-b8f7487cf-cgkdl"]
Oct 31 01:50:27.159070 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.158739    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chdbd\" (UniqueName: \"kubernetes.io/projected/2708e1b7-59fe-45a0-8b4e-320e8a365b17-kube-api-access-chdbd\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:27.159070 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.158768    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxdc8\" (UniqueName: \"kubernetes.io/projected/7b03b560-9699-4e2c-8970-5e7cf56c1cf9-kube-api-access-dxdc8\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:27.162862 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.162835    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-676801-84b8cddcf-fm7th"]
Oct 31 01:50:27.170274 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:27.170237    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-676801-84b8cddcf-fm7th"]
Oct 31 01:50:28.091138 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:28.091105    5650 generic.go:356] "Generic (PLEG): container finished" podID="e1f59db2-d04e-4e96-bf12-5a7d3f18eb99" containerID="2f447df343bf3d68976ec259c95615222060b10afd976e1cefaabe5d3ddcf293" exitCode=0
Oct 31 01:50:28.091618 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:28.091179    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td" event={"ID":"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99","Type":"ContainerDied","Data":"2f447df343bf3d68976ec259c95615222060b10afd976e1cefaabe5d3ddcf293"}
Oct 31 01:50:28.091990 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:28.091965    5650 scope.go:122] "RemoveContainer" containerID="2f447df343bf3d68976ec259c95615222060b10afd976e1cefaabe5d3ddcf293"
Oct 31 01:50:28.093891 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:28.093856    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:50:28.094021 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:28.093934    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:50:28.094021 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:28.093973    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:50:28.125571 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:28.125529    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2708e1b7-59fe-45a0-8b4e-320e8a365b17" path="/var/lib/kubelet/pods/2708e1b7-59fe-45a0-8b4e-320e8a365b17/volumes"
Oct 31 01:50:28.126234 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:28.126213    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b03b560-9699-4e2c-8970-5e7cf56c1cf9" path="/var/lib/kubelet/pods/7b03b560-9699-4e2c-8970-5e7cf56c1cf9/volumes"
Oct 31 01:50:28.558023 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:50:28.557986    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:50:29.126117 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:29.126075    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td" event={"ID":"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99","Type":"ContainerStarted","Data":"5fae99081b3a102413ea91fc4d1a179fd86b3e1e3516cbe35c37748cebad01e0"}
Oct 31 01:50:29.529563 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:29.529524    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:50:29.738698 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:29.738659    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:50:29.739873 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:29.739848    5650 patch_prober.go:29] interesting pod/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 31 01:50:29.739992 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:29.739908    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td" podUID="e1f59db2-d04e-4e96-bf12-5a7d3f18eb99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 31 01:50:30.739031 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:30.738995    5650 patch_prober.go:29] interesting pod/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 31 01:50:30.739427 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:30.739050    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td" podUID="e1f59db2-d04e-4e96-bf12-5a7d3f18eb99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 31 01:50:31.156200 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.156126    5650 generic.go:356] "Generic (PLEG): container finished" podID="868f691d-84c7-4301-b42f-3a314ffcdb0b" containerID="b0425641d7c3e62116c31d4cf3feca2ef23c72081857bdb6c75170da706a43d0" exitCode=0
Oct 31 01:50:31.157087 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.157065    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2" event={"ID":"868f691d-84c7-4301-b42f-3a314ffcdb0b","Type":"ContainerDied","Data":"b0425641d7c3e62116c31d4cf3feca2ef23c72081857bdb6c75170da706a43d0"}
Oct 31 01:50:31.157171 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.157143    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2" event={"ID":"868f691d-84c7-4301-b42f-3a314ffcdb0b","Type":"ContainerDied","Data":"9f61bade8b1c457a85a57a97864361d97c10ad4042b91f1fd0d0ff760e5f2bf4"}
Oct 31 01:50:31.157171 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.157161    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f61bade8b1c457a85a57a97864361d97c10ad4042b91f1fd0d0ff760e5f2bf4"
Oct 31 01:50:31.157246 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.157174    5650 scope.go:122] "RemoveContainer" containerID="7d2c5aa798f3989219b918d5ba62db13055c9cb10d601bba4a8940b89640e904"
Oct 31 01:50:31.167033 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.167004    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:50:31.283518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.277678    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-data\") pod \"868f691d-84c7-4301-b42f-3a314ffcdb0b\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") "
Oct 31 01:50:31.283518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.277754    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/868f691d-84c7-4301-b42f-3a314ffcdb0b-kube-api-access-v845r\" (UniqueName: \"kubernetes.io/projected/868f691d-84c7-4301-b42f-3a314ffcdb0b-kube-api-access-v845r\") pod \"868f691d-84c7-4301-b42f-3a314ffcdb0b\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") "
Oct 31 01:50:31.283518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.277797    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-podinfo\") pod \"868f691d-84c7-4301-b42f-3a314ffcdb0b\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") "
Oct 31 01:50:31.283518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.277825    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-credential-socket\") pod \"868f691d-84c7-4301-b42f-3a314ffcdb0b\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") "
Oct 31 01:50:31.283518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.277852    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-token\" (UniqueName: \"kubernetes.io/projected/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-token\") pod \"868f691d-84c7-4301-b42f-3a314ffcdb0b\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") "
Oct 31 01:50:31.283518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.277971    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/868f691d-84c7-4301-b42f-3a314ffcdb0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/868f691d-84c7-4301-b42f-3a314ffcdb0b-istiod-ca-cert\") pod \"868f691d-84c7-4301-b42f-3a314ffcdb0b\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") "
Oct 31 01:50:31.283518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.278125    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-workload-certs\") pod \"868f691d-84c7-4301-b42f-3a314ffcdb0b\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") "
Oct 31 01:50:31.283518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.278190    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-envoy\") pod \"868f691d-84c7-4301-b42f-3a314ffcdb0b\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") "
Oct 31 01:50:31.283518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.278226    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-workload-socket\") pod \"868f691d-84c7-4301-b42f-3a314ffcdb0b\" (UID: \"868f691d-84c7-4301-b42f-3a314ffcdb0b\") "
Oct 31 01:50:31.283518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.279434    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-data" pod "868f691d-84c7-4301-b42f-3a314ffcdb0b" (UID: "868f691d-84c7-4301-b42f-3a314ffcdb0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:31.292048 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.291732    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-credential-socket" pod "868f691d-84c7-4301-b42f-3a314ffcdb0b" (UID: "868f691d-84c7-4301-b42f-3a314ffcdb0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:31.311664 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.292217    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/868f691d-84c7-4301-b42f-3a314ffcdb0b-kube-api-access-v845r" pod "868f691d-84c7-4301-b42f-3a314ffcdb0b" (UID: "868f691d-84c7-4301-b42f-3a314ffcdb0b"). InnerVolumeSpecName "kube-api-access-v845r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:31.311664 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.297535    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-workload-socket" pod "868f691d-84c7-4301-b42f-3a314ffcdb0b" (UID: "868f691d-84c7-4301-b42f-3a314ffcdb0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:31.311664 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.298632    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-workload-certs" pod "868f691d-84c7-4301-b42f-3a314ffcdb0b" (UID: "868f691d-84c7-4301-b42f-3a314ffcdb0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:31.311664 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.299364    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/868f691d-84c7-4301-b42f-3a314ffcdb0b-istiod-ca-cert" pod "868f691d-84c7-4301-b42f-3a314ffcdb0b" (UID: "868f691d-84c7-4301-b42f-3a314ffcdb0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:31.336695 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.334466    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-token" pod "868f691d-84c7-4301-b42f-3a314ffcdb0b" (UID: "868f691d-84c7-4301-b42f-3a314ffcdb0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:31.395471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.390684    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:31.395471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.390730    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v845r\" (UniqueName: \"kubernetes.io/projected/868f691d-84c7-4301-b42f-3a314ffcdb0b-kube-api-access-v845r\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:31.395471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.390747    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:31.395471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.390763    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:31.395471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.390778    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/868f691d-84c7-4301-b42f-3a314ffcdb0b-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:31.395471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.390793    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:31.395471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.390817    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:31.470500 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.454735    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-envoy" pod "868f691d-84c7-4301-b42f-3a314ffcdb0b" (UID: "868f691d-84c7-4301-b42f-3a314ffcdb0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:31.471047 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.470987    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-podinfo" pod "868f691d-84c7-4301-b42f-3a314ffcdb0b" (UID: "868f691d-84c7-4301-b42f-3a314ffcdb0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:50:31.511513 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.511071    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:31.511513 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.511112    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/868f691d-84c7-4301-b42f-3a314ffcdb0b-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:31.739141 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.739107    5650 patch_prober.go:29] interesting pod/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 31 01:50:31.739544 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.739175    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td" podUID="e1f59db2-d04e-4e96-bf12-5a7d3f18eb99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 31 01:50:31.945588 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.945551    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8"]
Oct 31 01:50:31.946620 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.946600    5650 cpu_manager.go:414] "containerMap: removing container" podUID="7b03b560-9699-4e2c-8970-5e7cf56c1cf9" containerName="payload-processing"
Oct 31 01:50:31.946676 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.946623    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b03b560-9699-4e2c-8970-5e7cf56c1cf9" containerName="payload-processing"
Oct 31 01:50:31.946676 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.946639    5650 cpu_manager.go:414] "containerMap: removing container" podUID="868f691d-84c7-4301-b42f-3a314ffcdb0b" containerName="istio-proxy"
Oct 31 01:50:31.946676 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.946647    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="868f691d-84c7-4301-b42f-3a314ffcdb0b" containerName="istio-proxy"
Oct 31 01:50:31.946676 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.946670    5650 cpu_manager.go:414] "containerMap: removing container" podUID="2708e1b7-59fe-45a0-8b4e-320e8a365b17" containerName="payload-pre-processing"
Oct 31 01:50:31.946804 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.946679    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2708e1b7-59fe-45a0-8b4e-320e8a365b17" containerName="payload-pre-processing"
Oct 31 01:50:31.946804 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.946692    5650 cpu_manager.go:414] "containerMap: removing container" podUID="868f691d-84c7-4301-b42f-3a314ffcdb0b" containerName="istio-proxy"
Oct 31 01:50:31.946804 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.946699    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="868f691d-84c7-4301-b42f-3a314ffcdb0b" containerName="istio-proxy"
Oct 31 01:50:31.946910 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.946898    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b03b560-9699-4e2c-8970-5e7cf56c1cf9" containerName="payload-processing"
Oct 31 01:50:31.946946 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.946912    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="868f691d-84c7-4301-b42f-3a314ffcdb0b" containerName="istio-proxy"
Oct 31 01:50:31.946946 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.946926    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="2708e1b7-59fe-45a0-8b4e-320e8a365b17" containerName="payload-pre-processing"
Oct 31 01:50:31.946946 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.946935    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="868f691d-84c7-4301-b42f-3a314ffcdb0b" containerName="istio-proxy"
Oct 31 01:50:31.952561 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.952538    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8"
Oct 31 01:50:31.959959 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.959935    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8"]
Oct 31 01:50:31.962529 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:31.962499    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-90c4d1\""
Oct 31 01:50:32.033823 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.033737    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt"]
Oct 31 01:50:32.034351 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.034314    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkmhk\" (UniqueName: \"kubernetes.io/projected/eaf490d2-bacd-4a5b-bab2-58f20427a982-kube-api-access-rkmhk\") pod \"payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8\" (UID: \"eaf490d2-bacd-4a5b-bab2-58f20427a982\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8"
Oct 31 01:50:32.034626 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.034603    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaf490d2-bacd-4a5b-bab2-58f20427a982-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8\" (UID: \"eaf490d2-bacd-4a5b-bab2-58f20427a982\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8"
Oct 31 01:50:32.039435 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.039408    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt"]
Oct 31 01:50:32.039575 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.039422    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt"
Oct 31 01:50:32.048619 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.048593    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-90c4d1-dockercfg-nbmbq\""
Oct 31 01:50:32.148580 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.148504    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkmhk\" (UniqueName: \"kubernetes.io/projected/eaf490d2-bacd-4a5b-bab2-58f20427a982-kube-api-access-rkmhk\") pod \"payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8\" (UID: \"eaf490d2-bacd-4a5b-bab2-58f20427a982\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8"
Oct 31 01:50:32.148793 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.148724    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaf490d2-bacd-4a5b-bab2-58f20427a982-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8\" (UID: \"eaf490d2-bacd-4a5b-bab2-58f20427a982\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8"
Oct 31 01:50:32.172542 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.149833    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaf490d2-bacd-4a5b-bab2-58f20427a982-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8\" (UID: \"eaf490d2-bacd-4a5b-bab2-58f20427a982\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8"
Oct 31 01:50:32.213555 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.213503    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkmhk\" (UniqueName: \"kubernetes.io/projected/eaf490d2-bacd-4a5b-bab2-58f20427a982-kube-api-access-rkmhk\") pod \"payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8\" (UID: \"eaf490d2-bacd-4a5b-bab2-58f20427a982\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8"
Oct 31 01:50:32.255479 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.252027    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbnxk\" (UniqueName: \"kubernetes.io/projected/d2ace825-0835-4816-a346-7f9a72c3403e-kube-api-access-bbnxk\") pod \"payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt\" (UID: \"d2ace825-0835-4816-a346-7f9a72c3403e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt"
Oct 31 01:50:32.278623 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.277942    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ace825-0835-4816-a346-7f9a72c3403e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt\" (UID: \"d2ace825-0835-4816-a346-7f9a72c3403e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt"
Oct 31 01:50:32.286693 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.285475    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8"
Oct 31 01:50:32.387545 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.375401    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"
Oct 31 01:50:32.387545 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.380509    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ace825-0835-4816-a346-7f9a72c3403e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt\" (UID: \"d2ace825-0835-4816-a346-7f9a72c3403e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt"
Oct 31 01:50:32.387545 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.380625    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbnxk\" (UniqueName: \"kubernetes.io/projected/d2ace825-0835-4816-a346-7f9a72c3403e-kube-api-access-bbnxk\") pod \"payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt\" (UID: \"d2ace825-0835-4816-a346-7f9a72c3403e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt"
Oct 31 01:50:32.387545 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.381622    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ace825-0835-4816-a346-7f9a72c3403e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt\" (UID: \"d2ace825-0835-4816-a346-7f9a72c3403e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt"
Oct 31 01:50:32.461513 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.441482    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbnxk\" (UniqueName: \"kubernetes.io/projected/d2ace825-0835-4816-a346-7f9a72c3403e-kube-api-access-bbnxk\") pod \"payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt\" (UID: \"d2ace825-0835-4816-a346-7f9a72c3403e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt"
Oct 31 01:50:32.515478 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.514776    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"]
Oct 31 01:50:32.519472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.518081    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-cbbaec-openshift-default-686f7db48f-s76j2"]
Oct 31 01:50:32.655173 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.655135    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt"
Oct 31 01:50:32.730945 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.730898    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828"]
Oct 31 01:50:32.753338 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.741374    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828"
Oct 31 01:50:32.753338 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.743397    5650 patch_prober.go:29] interesting pod/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 31 01:50:32.753338 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.743474    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td" podUID="e1f59db2-d04e-4e96-bf12-5a7d3f18eb99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 31 01:50:32.753338 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.747280    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828"]
Oct 31 01:50:32.753338 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.747942    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-a779b3-dockercfg-sgtdc\""
Oct 31 01:50:32.753338 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.748103    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-a779b3\""
Oct 31 01:50:32.784310 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.784269    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8"]
Oct 31 01:50:32.786512 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:50:32.786430    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf490d2_bacd_4a5b_bab2_58f20427a982.slice/crio-7dda2e3f0bcddca218636318d10a098b805cb4a7396efd1c100bb8fec4f7bc15 WatchSource:0}: Error finding container 7dda2e3f0bcddca218636318d10a098b805cb4a7396efd1c100bb8fec4f7bc15: Status 404 returned error can't find the container with id 7dda2e3f0bcddca218636318d10a098b805cb4a7396efd1c100bb8fec4f7bc15
Oct 31 01:50:32.893468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.890898    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4zkm\" (UniqueName: \"kubernetes.io/projected/3e756d5a-ab71-453f-853a-2884bfe27fac-kube-api-access-l4zkm\") pod \"payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828\" (UID: \"3e756d5a-ab71-453f-853a-2884bfe27fac\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828"
Oct 31 01:50:32.893468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:32.891249    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e756d5a-ab71-453f-853a-2884bfe27fac-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828\" (UID: \"3e756d5a-ab71-453f-853a-2884bfe27fac\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828"
Oct 31 01:50:33.041475 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:33.021679    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e756d5a-ab71-453f-853a-2884bfe27fac-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828\" (UID: \"3e756d5a-ab71-453f-853a-2884bfe27fac\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828"
Oct 31 01:50:33.099804 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:33.099774    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4zkm\" (UniqueName: \"kubernetes.io/projected/3e756d5a-ab71-453f-853a-2884bfe27fac-kube-api-access-l4zkm\") pod \"payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828\" (UID: \"3e756d5a-ab71-453f-853a-2884bfe27fac\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828"
Oct 31 01:50:33.111670 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:33.099609    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e756d5a-ab71-453f-853a-2884bfe27fac-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828\" (UID: \"3e756d5a-ab71-453f-853a-2884bfe27fac\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828"
Oct 31 01:50:33.238480 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:33.237927    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4zkm\" (UniqueName: \"kubernetes.io/projected/3e756d5a-ab71-453f-853a-2884bfe27fac-kube-api-access-l4zkm\") pod \"payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828\" (UID: \"3e756d5a-ab71-453f-853a-2884bfe27fac\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828"
Oct 31 01:50:33.246018 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:33.245984    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt"]
Oct 31 01:50:33.374367 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:33.374122    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828"
Oct 31 01:50:33.390988 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:33.390961    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt" event={"ID":"d2ace825-0835-4816-a346-7f9a72c3403e","Type":"ContainerStarted","Data":"593c0fc2ee0078f5ce7a50072daa764c438ad44c89578beea786c66620a353ef"}
Oct 31 01:50:33.393508 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:33.393482    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8" event={"ID":"eaf490d2-bacd-4a5b-bab2-58f20427a982","Type":"ContainerStarted","Data":"7779ff426dd0f32f9fba856df891aae9b36f64c00fbe47ec8d5e93bbde417a68"}
Oct 31 01:50:33.393685 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:33.393666    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8" event={"ID":"eaf490d2-bacd-4a5b-bab2-58f20427a982","Type":"ContainerStarted","Data":"7dda2e3f0bcddca218636318d10a098b805cb4a7396efd1c100bb8fec4f7bc15"}
Oct 31 01:50:33.394676 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:33.394655    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8"
Oct 31 01:50:33.434806 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:33.434750    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8" podStartSLOduration=2.434732402 podStartE2EDuration="2.434732402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:50:33.423886335 +0000 UTC m=+1789.427600627" watchObservedRunningTime="2026-10-31 01:50:33.434732402 +0000 UTC m=+1789.438446708"
Oct 31 01:50:33.761530 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:33.750893    5650 patch_prober.go:29] interesting pod/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 31 01:50:33.761530 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:33.750954    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td" podUID="e1f59db2-d04e-4e96-bf12-5a7d3f18eb99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 31 01:50:33.770000 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:33.767210    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828"]
Oct 31 01:50:34.165066 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:34.164937    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="868f691d-84c7-4301-b42f-3a314ffcdb0b" path="/var/lib/kubelet/pods/868f691d-84c7-4301-b42f-3a314ffcdb0b/volumes"
Oct 31 01:50:34.539092 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:34.539048    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828" event={"ID":"3e756d5a-ab71-453f-853a-2884bfe27fac","Type":"ContainerStarted","Data":"8926b92452cc4bf50e48e06b15422719840b4938ec31dac479aece0bf67478b9"}
Oct 31 01:50:34.539392 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:34.539366    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828" event={"ID":"3e756d5a-ab71-453f-853a-2884bfe27fac","Type":"ContainerStarted","Data":"64f434ad7f3e1742587b59204d365ee6df34a8a2fa93b7ccd8d24acf6a5304c5"}
Oct 31 01:50:34.540746 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:34.540721    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828"
Oct 31 01:50:34.557394 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:34.557362    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt" event={"ID":"d2ace825-0835-4816-a346-7f9a72c3403e","Type":"ContainerStarted","Data":"480221c1bebd6bac282511539c8caf6be93bc349bbf8d0e73493d06e89c82715"}
Oct 31 01:50:34.557677 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:34.557659    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt"
Oct 31 01:50:34.627303 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:34.627236    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt" podStartSLOduration=3.627215464 podStartE2EDuration="3.627215464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:50:34.621199988 +0000 UTC m=+1790.624914281" watchObservedRunningTime="2026-10-31 01:50:34.627215464 +0000 UTC m=+1790.630929758"
Oct 31 01:50:34.627686 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:34.627643    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828" podStartSLOduration=2.627625804 podStartE2EDuration="2.627625804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:50:34.586284187 +0000 UTC m=+1790.589998480" watchObservedRunningTime="2026-10-31 01:50:34.627625804 +0000 UTC m=+1790.631340096"
Oct 31 01:50:34.738422 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:34.738375    5650 patch_prober.go:29] interesting pod/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 31 01:50:34.738632 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:34.738440    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td" podUID="e1f59db2-d04e-4e96-bf12-5a7d3f18eb99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 31 01:50:35.744284 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:35.744248    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:50:35.746273 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:35.746245    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:50:42.465851 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:42.465805    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"]
Oct 31 01:50:42.466407 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:42.466080    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8" podUID="da0a52af-42a1-45d5-ab84-6696241d5489" containerName="istio-proxy" containerID="cri-o://2b5a02e7e7c0411d921fb61540a74f4fbf32fca53482c1b072d1ec48b6b90004" gracePeriod=30
Oct 31 01:50:42.472600 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:42.472549    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"]
Oct 31 01:50:42.472877 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:42.472817    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4" podUID="d2118bab-9cba-42f7-814d-730046ef72a2" containerName="istio-proxy" containerID="cri-o://bc9360ecec41e1006c1f672e65ab2c886e14eff66250d4cb2303669173966a91" gracePeriod=30
Oct 31 01:50:42.478989 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:42.478957    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"]
Oct 31 01:50:42.479224 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:42.479185    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx" podUID="ebc90b15-cef7-4483-a8bb-1a33950f9fa6" containerName="istio-proxy" containerID="cri-o://d825246430999b2486f70322c978e1ee5d7b23a345cc29a5910929760122a431" gracePeriod=30
Oct 31 01:50:44.559282 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:44.559249    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8"
Oct 31 01:50:44.650048 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:44.650013    5650 scope.go:122] "RemoveContainer" containerID="f60e22db9a1494bb66b85c62017c24dfb5c5b05accb9b440ed326d534ff6efd7"
Oct 31 01:50:44.666625 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:44.664736    5650 scope.go:122] "RemoveContainer" containerID="37d967cd3f2e322e95aeb7e17daf9bc05dac826cce952fef71a4b9fdc748662e"
Oct 31 01:50:46.573472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:46.573407    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt"
Oct 31 01:50:46.574514 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:46.574497    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828"
Oct 31 01:50:48.073255 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.073217    5650 generic.go:356] "Generic (PLEG): container finished" podID="d2118bab-9cba-42f7-814d-730046ef72a2" containerID="bc9360ecec41e1006c1f672e65ab2c886e14eff66250d4cb2303669173966a91" exitCode=0
Oct 31 01:50:48.073688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.073426    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4" event={"ID":"d2118bab-9cba-42f7-814d-730046ef72a2","Type":"ContainerDied","Data":"bc9360ecec41e1006c1f672e65ab2c886e14eff66250d4cb2303669173966a91"}
Oct 31 01:50:48.075600 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.075492    5650 generic.go:356] "Generic (PLEG): container finished" podID="da0a52af-42a1-45d5-ab84-6696241d5489" containerID="2b5a02e7e7c0411d921fb61540a74f4fbf32fca53482c1b072d1ec48b6b90004" exitCode=0
Oct 31 01:50:48.075694 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.075670    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8" event={"ID":"da0a52af-42a1-45d5-ab84-6696241d5489","Type":"ContainerDied","Data":"2b5a02e7e7c0411d921fb61540a74f4fbf32fca53482c1b072d1ec48b6b90004"}
Oct 31 01:50:48.077987 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.077962    5650 generic.go:356] "Generic (PLEG): container finished" podID="ebc90b15-cef7-4483-a8bb-1a33950f9fa6" containerID="d825246430999b2486f70322c978e1ee5d7b23a345cc29a5910929760122a431" exitCode=0
Oct 31 01:50:48.078086 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.078042    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx" event={"ID":"ebc90b15-cef7-4483-a8bb-1a33950f9fa6","Type":"ContainerDied","Data":"d825246430999b2486f70322c978e1ee5d7b23a345cc29a5910929760122a431"}
Oct 31 01:50:48.088948 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.088913    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:50:48.196850 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.189087    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-istio-envoy\") pod \"da0a52af-42a1-45d5-ab84-6696241d5489\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") "
Oct 31 01:50:48.196850 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.190209    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da0a52af-42a1-45d5-ab84-6696241d5489-kube-api-access-4zx96\" (UniqueName: \"kubernetes.io/projected/da0a52af-42a1-45d5-ab84-6696241d5489-kube-api-access-4zx96\") pod \"da0a52af-42a1-45d5-ab84-6696241d5489\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") "
Oct 31 01:50:48.223521 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.207609    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-istio-envoy" pod "da0a52af-42a1-45d5-ab84-6696241d5489" (UID: "da0a52af-42a1-45d5-ab84-6696241d5489"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:48.223521 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.218727    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da0a52af-42a1-45d5-ab84-6696241d5489-istio-token\" (UniqueName: \"kubernetes.io/projected/da0a52af-42a1-45d5-ab84-6696241d5489-istio-token\") pod \"da0a52af-42a1-45d5-ab84-6696241d5489\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") "
Oct 31 01:50:48.223521 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.218797    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da0a52af-42a1-45d5-ab84-6696241d5489-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da0a52af-42a1-45d5-ab84-6696241d5489-istio-podinfo\") pod \"da0a52af-42a1-45d5-ab84-6696241d5489\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") "
Oct 31 01:50:48.243556 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.243509    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-credential-socket\") pod \"da0a52af-42a1-45d5-ab84-6696241d5489\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") "
Oct 31 01:50:48.243746 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.243640    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da0a52af-42a1-45d5-ab84-6696241d5489-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da0a52af-42a1-45d5-ab84-6696241d5489-istiod-ca-cert\") pod \"da0a52af-42a1-45d5-ab84-6696241d5489\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") "
Oct 31 01:50:48.243829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.243786    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-istio-data\") pod \"da0a52af-42a1-45d5-ab84-6696241d5489\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") "
Oct 31 01:50:48.243917 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.243890    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-workload-socket\") pod \"da0a52af-42a1-45d5-ab84-6696241d5489\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") "
Oct 31 01:50:48.243981 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.243929    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-workload-certs\") pod \"da0a52af-42a1-45d5-ab84-6696241d5489\" (UID: \"da0a52af-42a1-45d5-ab84-6696241d5489\") "
Oct 31 01:50:48.244125 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.244088    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-credential-socket" pod "da0a52af-42a1-45d5-ab84-6696241d5489" (UID: "da0a52af-42a1-45d5-ab84-6696241d5489"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:48.244481 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.244436    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-istio-data" pod "da0a52af-42a1-45d5-ab84-6696241d5489" (UID: "da0a52af-42a1-45d5-ab84-6696241d5489"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:48.244601 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.244488    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-workload-certs" pod "da0a52af-42a1-45d5-ab84-6696241d5489" (UID: "da0a52af-42a1-45d5-ab84-6696241d5489"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:48.244601 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.244170    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da0a52af-42a1-45d5-ab84-6696241d5489-istiod-ca-cert" pod "da0a52af-42a1-45d5-ab84-6696241d5489" (UID: "da0a52af-42a1-45d5-ab84-6696241d5489"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:48.245462 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.245415    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-workload-socket" pod "da0a52af-42a1-45d5-ab84-6696241d5489" (UID: "da0a52af-42a1-45d5-ab84-6696241d5489"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:48.246207 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.246162    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.246207 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.246193    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.246362 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.246208    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.246362 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.246223    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.246362 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.246237    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da0a52af-42a1-45d5-ab84-6696241d5489-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.246362 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.246253    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da0a52af-42a1-45d5-ab84-6696241d5489-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.248171 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.247975    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da0a52af-42a1-45d5-ab84-6696241d5489-kube-api-access-4zx96" pod "da0a52af-42a1-45d5-ab84-6696241d5489" (UID: "da0a52af-42a1-45d5-ab84-6696241d5489"). InnerVolumeSpecName "kube-api-access-4zx96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:48.248750 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.248721    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da0a52af-42a1-45d5-ab84-6696241d5489-istio-token" pod "da0a52af-42a1-45d5-ab84-6696241d5489" (UID: "da0a52af-42a1-45d5-ab84-6696241d5489"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:48.261119 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.260618    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da0a52af-42a1-45d5-ab84-6696241d5489-istio-podinfo" pod "da0a52af-42a1-45d5-ab84-6696241d5489" (UID: "da0a52af-42a1-45d5-ab84-6696241d5489"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:50:48.273140 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.273099    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:50:48.282084 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.282058    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:50:48.347063 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.346972    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2118bab-9cba-42f7-814d-730046ef72a2-kube-api-access-7ds9z\" (UniqueName: \"kubernetes.io/projected/d2118bab-9cba-42f7-814d-730046ef72a2-kube-api-access-7ds9z\") pod \"d2118bab-9cba-42f7-814d-730046ef72a2\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") "
Oct 31 01:50:48.347063 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.347045    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-credential-socket\") pod \"d2118bab-9cba-42f7-814d-730046ef72a2\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") "
Oct 31 01:50:48.347316 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.347080    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-data\") pod \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") "
Oct 31 01:50:48.347316 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.347100    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-credential-socket\") pod \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") "
Oct 31 01:50:48.347316 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.347125    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-podinfo\") pod \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") "
Oct 31 01:50:48.347316 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.347146    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-workload-socket\") pod \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") "
Oct 31 01:50:48.347316 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.347180    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istiod-ca-cert\") pod \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") "
Oct 31 01:50:48.347316 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.347199    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-token\" (UniqueName: \"kubernetes.io/projected/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-token\") pod \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") "
Oct 31 01:50:48.347316 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.347221    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-workload-socket\") pod \"d2118bab-9cba-42f7-814d-730046ef72a2\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") "
Oct 31 01:50:48.347316 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.347267    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-envoy\") pod \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") "
Oct 31 01:50:48.347316 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.347287    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-istio-data\") pod \"d2118bab-9cba-42f7-814d-730046ef72a2\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") "
Oct 31 01:50:48.347842 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.347337    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-workload-certs\") pod \"d2118bab-9cba-42f7-814d-730046ef72a2\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") "
Oct 31 01:50:48.347842 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.347355    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-workload-certs\") pod \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") "
Oct 31 01:50:48.347842 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.347375    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2118bab-9cba-42f7-814d-730046ef72a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2118bab-9cba-42f7-814d-730046ef72a2-istiod-ca-cert\") pod \"d2118bab-9cba-42f7-814d-730046ef72a2\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") "
Oct 31 01:50:48.347842 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.347395    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-kube-api-access-d2tt8\" (UniqueName: \"kubernetes.io/projected/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-kube-api-access-d2tt8\") pod \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\" (UID: \"ebc90b15-cef7-4483-a8bb-1a33950f9fa6\") "
Oct 31 01:50:48.347842 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.347540    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2118bab-9cba-42f7-814d-730046ef72a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2118bab-9cba-42f7-814d-730046ef72a2-istio-podinfo\") pod \"d2118bab-9cba-42f7-814d-730046ef72a2\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") "
Oct 31 01:50:48.347842 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.347577    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2118bab-9cba-42f7-814d-730046ef72a2-istio-token\" (UniqueName: \"kubernetes.io/projected/d2118bab-9cba-42f7-814d-730046ef72a2-istio-token\") pod \"d2118bab-9cba-42f7-814d-730046ef72a2\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") "
Oct 31 01:50:48.347842 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.347644    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-istio-envoy\") pod \"d2118bab-9cba-42f7-814d-730046ef72a2\" (UID: \"d2118bab-9cba-42f7-814d-730046ef72a2\") "
Oct 31 01:50:48.348201 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.348122    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zx96\" (UniqueName: \"kubernetes.io/projected/da0a52af-42a1-45d5-ab84-6696241d5489-kube-api-access-4zx96\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.348201 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.348137    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da0a52af-42a1-45d5-ab84-6696241d5489-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.348201 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.348146    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da0a52af-42a1-45d5-ab84-6696241d5489-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.349085 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.348648    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-workload-socket" pod "d2118bab-9cba-42f7-814d-730046ef72a2" (UID: "d2118bab-9cba-42f7-814d-730046ef72a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:48.349085 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.348721    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-workload-socket" pod "ebc90b15-cef7-4483-a8bb-1a33950f9fa6" (UID: "ebc90b15-cef7-4483-a8bb-1a33950f9fa6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:48.349085 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.349018    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istiod-ca-cert" pod "ebc90b15-cef7-4483-a8bb-1a33950f9fa6" (UID: "ebc90b15-cef7-4483-a8bb-1a33950f9fa6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:48.349578 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.349555    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-credential-socket" pod "ebc90b15-cef7-4483-a8bb-1a33950f9fa6" (UID: "ebc90b15-cef7-4483-a8bb-1a33950f9fa6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:48.350037 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.350011    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-credential-socket" pod "d2118bab-9cba-42f7-814d-730046ef72a2" (UID: "d2118bab-9cba-42f7-814d-730046ef72a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:48.350215 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.350192    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-data" pod "ebc90b15-cef7-4483-a8bb-1a33950f9fa6" (UID: "ebc90b15-cef7-4483-a8bb-1a33950f9fa6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:48.350794 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.350626    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-workload-certs" pod "ebc90b15-cef7-4483-a8bb-1a33950f9fa6" (UID: "ebc90b15-cef7-4483-a8bb-1a33950f9fa6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:48.350794 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.350643    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-istio-data" pod "d2118bab-9cba-42f7-814d-730046ef72a2" (UID: "d2118bab-9cba-42f7-814d-730046ef72a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:48.350956 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.350927    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2118bab-9cba-42f7-814d-730046ef72a2-istiod-ca-cert" pod "d2118bab-9cba-42f7-814d-730046ef72a2" (UID: "d2118bab-9cba-42f7-814d-730046ef72a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:48.351341 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.351070    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-workload-certs" pod "d2118bab-9cba-42f7-814d-730046ef72a2" (UID: "d2118bab-9cba-42f7-814d-730046ef72a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:48.352564 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.352528    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-podinfo" pod "ebc90b15-cef7-4483-a8bb-1a33950f9fa6" (UID: "ebc90b15-cef7-4483-a8bb-1a33950f9fa6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:50:48.354815 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.354777    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2118bab-9cba-42f7-814d-730046ef72a2-istio-podinfo" pod "d2118bab-9cba-42f7-814d-730046ef72a2" (UID: "d2118bab-9cba-42f7-814d-730046ef72a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:50:48.356131 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.356106    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2118bab-9cba-42f7-814d-730046ef72a2-istio-token" pod "d2118bab-9cba-42f7-814d-730046ef72a2" (UID: "d2118bab-9cba-42f7-814d-730046ef72a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:48.356703 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.356683    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-envoy" pod "ebc90b15-cef7-4483-a8bb-1a33950f9fa6" (UID: "ebc90b15-cef7-4483-a8bb-1a33950f9fa6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:48.356840 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.356824    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2118bab-9cba-42f7-814d-730046ef72a2-kube-api-access-7ds9z" pod "d2118bab-9cba-42f7-814d-730046ef72a2" (UID: "d2118bab-9cba-42f7-814d-730046ef72a2"). InnerVolumeSpecName "kube-api-access-7ds9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:48.356914 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.356859    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-token" pod "ebc90b15-cef7-4483-a8bb-1a33950f9fa6" (UID: "ebc90b15-cef7-4483-a8bb-1a33950f9fa6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:48.356971 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.356910    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-istio-envoy" pod "d2118bab-9cba-42f7-814d-730046ef72a2" (UID: "d2118bab-9cba-42f7-814d-730046ef72a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:48.358392 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.358371    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-kube-api-access-d2tt8" pod "ebc90b15-cef7-4483-a8bb-1a33950f9fa6" (UID: "ebc90b15-cef7-4483-a8bb-1a33950f9fa6"). InnerVolumeSpecName "kube-api-access-d2tt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:48.450162 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.450120    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.450162 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.450157    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.450162 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.450167    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.450409 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.450179    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.450409 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.450188    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.450409 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.450196    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.450409 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.450205    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.450409 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.450214    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.450409 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.450222    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.450409 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.450230    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.450409 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.450238    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2118bab-9cba-42f7-814d-730046ef72a2-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.450409 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.450247    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2tt8\" (UniqueName: \"kubernetes.io/projected/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-kube-api-access-d2tt8\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.450409 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.450255    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2118bab-9cba-42f7-814d-730046ef72a2-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.450409 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.450264    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2118bab-9cba-42f7-814d-730046ef72a2-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.450409 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.450272    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.450409 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.450280    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ds9z\" (UniqueName: \"kubernetes.io/projected/d2118bab-9cba-42f7-814d-730046ef72a2-kube-api-access-7ds9z\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.450409 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.450289    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2118bab-9cba-42f7-814d-730046ef72a2-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.450409 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:48.450297    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebc90b15-cef7-4483-a8bb-1a33950f9fa6-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:49.088276 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:49.088236    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"
Oct 31 01:50:49.088276 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:49.088262    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx" event={"ID":"ebc90b15-cef7-4483-a8bb-1a33950f9fa6","Type":"ContainerDied","Data":"32fbbb9f634534f3892f0aac9e883e3a0b82e477be6541d29002f8a0e3c9f0bc"}
Oct 31 01:50:49.088843 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:49.088326    5650 scope.go:122] "RemoveContainer" containerID="d825246430999b2486f70322c978e1ee5d7b23a345cc29a5910929760122a431"
Oct 31 01:50:49.089856 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:49.089836    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"
Oct 31 01:50:49.089931 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:49.089870    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4" event={"ID":"d2118bab-9cba-42f7-814d-730046ef72a2","Type":"ContainerDied","Data":"8a59830824ec5daf23a7942ba91d7fd24c0c4b5b7d220fd0d5ef112d5fd8b290"}
Oct 31 01:50:49.091499 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:49.091476    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8" event={"ID":"da0a52af-42a1-45d5-ab84-6696241d5489","Type":"ContainerDied","Data":"7b83ed2eb5e7940f583b3c87663b3d35239ed955044d73f46623060dc912652c"}
Oct 31 01:50:49.091637 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:49.091513    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"
Oct 31 01:50:49.125656 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:49.125625    5650 scope.go:122] "RemoveContainer" containerID="bc9360ecec41e1006c1f672e65ab2c886e14eff66250d4cb2303669173966a91"
Oct 31 01:50:49.136971 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:49.136945    5650 scope.go:122] "RemoveContainer" containerID="2b5a02e7e7c0411d921fb61540a74f4fbf32fca53482c1b072d1ec48b6b90004"
Oct 31 01:50:49.148724 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:49.148697    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"]
Oct 31 01:50:49.158052 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:49.158021    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-rtjb4"]
Oct 31 01:50:49.171801 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:49.171769    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"]
Oct 31 01:50:49.176080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:49.176054    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-r8lmx"]
Oct 31 01:50:49.191352 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:49.191192    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"]
Oct 31 01:50:49.197980 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:49.197950    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sxwm8"]
Oct 31 01:50:50.057347 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.056819    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6"]
Oct 31 01:50:50.061216 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.061165    5650 cpu_manager.go:414] "containerMap: removing container" podUID="d2118bab-9cba-42f7-814d-730046ef72a2" containerName="istio-proxy"
Oct 31 01:50:50.061216 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.061203    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2118bab-9cba-42f7-814d-730046ef72a2" containerName="istio-proxy"
Oct 31 01:50:50.061399 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.061235    5650 cpu_manager.go:414] "containerMap: removing container" podUID="ebc90b15-cef7-4483-a8bb-1a33950f9fa6" containerName="istio-proxy"
Oct 31 01:50:50.061502 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.061413    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebc90b15-cef7-4483-a8bb-1a33950f9fa6" containerName="istio-proxy"
Oct 31 01:50:50.061546 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.061508    5650 cpu_manager.go:414] "containerMap: removing container" podUID="da0a52af-42a1-45d5-ab84-6696241d5489" containerName="istio-proxy"
Oct 31 01:50:50.061546 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.061519    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da0a52af-42a1-45d5-ab84-6696241d5489" containerName="istio-proxy"
Oct 31 01:50:50.061784 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.061711    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="da0a52af-42a1-45d5-ab84-6696241d5489" containerName="istio-proxy"
Oct 31 01:50:50.061784 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.061737    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2118bab-9cba-42f7-814d-730046ef72a2" containerName="istio-proxy"
Oct 31 01:50:50.061784 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.061748    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebc90b15-cef7-4483-a8bb-1a33950f9fa6" containerName="istio-proxy"
Oct 31 01:50:50.098254 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.092015    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6"]
Oct 31 01:50:50.098254 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.092342    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6"
Oct 31 01:50:50.106576 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.099996    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:50:50.106576 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.100161    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:50:50.140239 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.140197    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2118bab-9cba-42f7-814d-730046ef72a2" path="/var/lib/kubelet/pods/d2118bab-9cba-42f7-814d-730046ef72a2/volumes"
Oct 31 01:50:50.140909 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.140885    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da0a52af-42a1-45d5-ab84-6696241d5489" path="/var/lib/kubelet/pods/da0a52af-42a1-45d5-ab84-6696241d5489/volumes"
Oct 31 01:50:50.141353 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.141336    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebc90b15-cef7-4483-a8bb-1a33950f9fa6" path="/var/lib/kubelet/pods/ebc90b15-cef7-4483-a8bb-1a33950f9fa6/volumes"
Oct 31 01:50:50.206915 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.206875    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4da93e-cf46-4af6-9255-1c1e150eb67c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6\" (UID: \"0f4da93e-cf46-4af6-9255-1c1e150eb67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6"
Oct 31 01:50:50.207101 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.207080    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k8r5\" (UniqueName: \"kubernetes.io/projected/0f4da93e-cf46-4af6-9255-1c1e150eb67c-kube-api-access-8k8r5\") pod \"maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6\" (UID: \"0f4da93e-cf46-4af6-9255-1c1e150eb67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6"
Oct 31 01:50:50.207315 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.207287    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4da93e-cf46-4af6-9255-1c1e150eb67c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6\" (UID: \"0f4da93e-cf46-4af6-9255-1c1e150eb67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6"
Oct 31 01:50:50.308969 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.308864    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4da93e-cf46-4af6-9255-1c1e150eb67c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6\" (UID: \"0f4da93e-cf46-4af6-9255-1c1e150eb67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6"
Oct 31 01:50:50.309148 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.309012    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k8r5\" (UniqueName: \"kubernetes.io/projected/0f4da93e-cf46-4af6-9255-1c1e150eb67c-kube-api-access-8k8r5\") pod \"maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6\" (UID: \"0f4da93e-cf46-4af6-9255-1c1e150eb67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6"
Oct 31 01:50:50.309148 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.309106    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4da93e-cf46-4af6-9255-1c1e150eb67c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6\" (UID: \"0f4da93e-cf46-4af6-9255-1c1e150eb67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6"
Oct 31 01:50:50.309676 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.309647    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4da93e-cf46-4af6-9255-1c1e150eb67c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6\" (UID: \"0f4da93e-cf46-4af6-9255-1c1e150eb67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6"
Oct 31 01:50:50.312241 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.312202    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4da93e-cf46-4af6-9255-1c1e150eb67c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6\" (UID: \"0f4da93e-cf46-4af6-9255-1c1e150eb67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6"
Oct 31 01:50:50.323811 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.323780    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k8r5\" (UniqueName: \"kubernetes.io/projected/0f4da93e-cf46-4af6-9255-1c1e150eb67c-kube-api-access-8k8r5\") pod \"maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6\" (UID: \"0f4da93e-cf46-4af6-9255-1c1e150eb67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6"
Oct 31 01:50:50.427822 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.427786    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6"
Oct 31 01:50:50.717818 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:50.717785    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6"]
Oct 31 01:50:50.717977 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:50:50.717957    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4da93e_cf46_4af6_9255_1c1e150eb67c.slice/crio-c54184d39c821f2aec71cafcd9bcc7c6295f2bdb4b09d4ebe69b7e268d4ca718 WatchSource:0}: Error finding container c54184d39c821f2aec71cafcd9bcc7c6295f2bdb4b09d4ebe69b7e268d4ca718: Status 404 returned error can't find the container with id c54184d39c821f2aec71cafcd9bcc7c6295f2bdb4b09d4ebe69b7e268d4ca718
Oct 31 01:50:51.154109 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:51.154070    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6" event={"ID":"0f4da93e-cf46-4af6-9255-1c1e150eb67c","Type":"ContainerStarted","Data":"b696cd6e32924d85482e642fdf4d1c31b42d014e9c73c802d4fdaccc9bbfeba5"}
Oct 31 01:50:51.154513 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:51.154133    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6" event={"ID":"0f4da93e-cf46-4af6-9255-1c1e150eb67c","Type":"ContainerStarted","Data":"c54184d39c821f2aec71cafcd9bcc7c6295f2bdb4b09d4ebe69b7e268d4ca718"}
Oct 31 01:50:51.181493 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:51.181416    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6" podStartSLOduration=1.18140151 podStartE2EDuration="1.18140151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:50:51.177126701 +0000 UTC m=+1807.180840973" watchObservedRunningTime="2026-10-31 01:50:51.18140151 +0000 UTC m=+1807.185115801"
Oct 31 01:50:52.164062 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:52.164030    5650 generic.go:356] "Generic (PLEG): container finished" podID="0f4da93e-cf46-4af6-9255-1c1e150eb67c" containerID="b696cd6e32924d85482e642fdf4d1c31b42d014e9c73c802d4fdaccc9bbfeba5" exitCode=0
Oct 31 01:50:52.164487 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:52.164115    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6" event={"ID":"0f4da93e-cf46-4af6-9255-1c1e150eb67c","Type":"ContainerDied","Data":"b696cd6e32924d85482e642fdf4d1c31b42d014e9c73c802d4fdaccc9bbfeba5"}
Oct 31 01:50:53.358896 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:53.358861    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6"
Oct 31 01:50:53.475396 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:53.471978    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f4da93e-cf46-4af6-9255-1c1e150eb67c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4da93e-cf46-4af6-9255-1c1e150eb67c-maas-api-openshift-ca\") pod \"0f4da93e-cf46-4af6-9255-1c1e150eb67c\" (UID: \"0f4da93e-cf46-4af6-9255-1c1e150eb67c\") "
Oct 31 01:50:53.475396 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:53.472060    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f4da93e-cf46-4af6-9255-1c1e150eb67c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4da93e-cf46-4af6-9255-1c1e150eb67c-maas-api-xks-ca\") pod \"0f4da93e-cf46-4af6-9255-1c1e150eb67c\" (UID: \"0f4da93e-cf46-4af6-9255-1c1e150eb67c\") "
Oct 31 01:50:53.475396 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:53.472138    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f4da93e-cf46-4af6-9255-1c1e150eb67c-kube-api-access-8k8r5\" (UniqueName: \"kubernetes.io/projected/0f4da93e-cf46-4af6-9255-1c1e150eb67c-kube-api-access-8k8r5\") pod \"0f4da93e-cf46-4af6-9255-1c1e150eb67c\" (UID: \"0f4da93e-cf46-4af6-9255-1c1e150eb67c\") "
Oct 31 01:50:53.475396 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:53.472420    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f4da93e-cf46-4af6-9255-1c1e150eb67c-maas-api-openshift-ca" pod "0f4da93e-cf46-4af6-9255-1c1e150eb67c" (UID: "0f4da93e-cf46-4af6-9255-1c1e150eb67c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:53.475396 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:53.472903    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4da93e-cf46-4af6-9255-1c1e150eb67c-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:53.475396 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:53.474372    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4da93e-cf46-4af6-9255-1c1e150eb67c-kube-api-access-8k8r5" pod "0f4da93e-cf46-4af6-9255-1c1e150eb67c" (UID: "0f4da93e-cf46-4af6-9255-1c1e150eb67c"). InnerVolumeSpecName "kube-api-access-8k8r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:53.475396 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:53.474394    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f4da93e-cf46-4af6-9255-1c1e150eb67c-maas-api-xks-ca" pod "0f4da93e-cf46-4af6-9255-1c1e150eb67c" (UID: "0f4da93e-cf46-4af6-9255-1c1e150eb67c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:50:53.682044 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:53.574504    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4da93e-cf46-4af6-9255-1c1e150eb67c-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:53.682044 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:53.574547    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k8r5\" (UniqueName: \"kubernetes.io/projected/0f4da93e-cf46-4af6-9255-1c1e150eb67c-kube-api-access-8k8r5\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:50:54.182570 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:54.182529    5650 generic.go:356] "Generic (PLEG): container finished" podID="dcbe8798-c240-4556-bc46-d28d7ffacef2" containerID="87d9887636d1211e30cfd8159074b990b8f029a0c3dc68a70ea34dd8ac4e0c01" exitCode=0
Oct 31 01:50:54.182749 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:54.182579    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn" event={"ID":"dcbe8798-c240-4556-bc46-d28d7ffacef2","Type":"ContainerDied","Data":"87d9887636d1211e30cfd8159074b990b8f029a0c3dc68a70ea34dd8ac4e0c01"}
Oct 31 01:50:54.183136 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:54.183117    5650 scope.go:122] "RemoveContainer" containerID="87d9887636d1211e30cfd8159074b990b8f029a0c3dc68a70ea34dd8ac4e0c01"
Oct 31 01:50:54.184342 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:54.184315    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6" event={"ID":"0f4da93e-cf46-4af6-9255-1c1e150eb67c","Type":"ContainerDied","Data":"c54184d39c821f2aec71cafcd9bcc7c6295f2bdb4b09d4ebe69b7e268d4ca718"}
Oct 31 01:50:54.184342 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:54.184332    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6"
Oct 31 01:50:54.184553 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:54.184379    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c54184d39c821f2aec71cafcd9bcc7c6295f2bdb4b09d4ebe69b7e268d4ca718"
Oct 31 01:50:54.185341 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:54.185309    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:50:54.185487 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:54.185394    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:50:54.185487 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:54.185477    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:50:55.196200 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:55.196106    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn" event={"ID":"dcbe8798-c240-4556-bc46-d28d7ffacef2","Type":"ContainerStarted","Data":"09ae3a15b3a6aee49a89246a03cb493cbd3f522219b5bbdd1e087c64c3685cd0"}
Oct 31 01:50:56.164254 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:56.164217    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:50:56.165567 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:56.165535    5650 patch_prober.go:29] interesting pod/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 31 01:50:56.165687 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:56.165590    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn" podUID="dcbe8798-c240-4556-bc46-d28d7ffacef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 31 01:50:57.164677 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:57.164638    5650 patch_prober.go:29] interesting pod/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 31 01:50:57.165106 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:57.164711    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn" podUID="dcbe8798-c240-4556-bc46-d28d7ffacef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 31 01:50:58.141136 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.141090    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"]
Oct 31 01:50:58.142193 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.142163    5650 cpu_manager.go:414] "containerMap: removing container" podUID="0f4da93e-cf46-4af6-9255-1c1e150eb67c" containerName="revoke-keys"
Oct 31 01:50:58.142193 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.142192    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f4da93e-cf46-4af6-9255-1c1e150eb67c" containerName="revoke-keys"
Oct 31 01:50:58.142408 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.142391    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f4da93e-cf46-4af6-9255-1c1e150eb67c" containerName="revoke-keys"
Oct 31 01:50:58.147141 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.147116    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.161380 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.161333    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"]
Oct 31 01:50:58.162634 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.162247    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-726dd1-openshift-default-dockercfg-nsl8w\""
Oct 31 01:50:58.163978 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.163952    5650 patch_prober.go:29] interesting pod/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 31 01:50:58.166035 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.165985    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn" podUID="dcbe8798-c240-4556-bc46-d28d7ffacef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 31 01:50:58.187845 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.187799    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-token\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.188053 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.187882    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-podinfo\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.188053 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.187928    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr9m9\" (UniqueName: \"kubernetes.io/projected/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-kube-api-access-rr9m9\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.188053 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.187975    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-workload-certs\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.188277 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.188247    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-credential-socket\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.188403 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.188384    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-workload-socket\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.188500 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.188482    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-data\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.188619 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.188584    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-envoy\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.188684 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.188658    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istiod-ca-cert\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.292515 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.292466    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-workload-certs\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.306182 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.306153    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-credential-socket\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.306492 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.305967    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-workload-certs\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.306660 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.306509    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-workload-socket\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.306812 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.306790    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-workload-socket\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.337619 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.306874    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-data\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.337996 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.337970    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-envoy\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.338230 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.338153    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istiod-ca-cert\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.338429 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.338409    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-token\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.338628 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.338609    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-podinfo\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.338767 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.338749    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr9m9\" (UniqueName: \"kubernetes.io/projected/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-kube-api-access-rr9m9\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.339369 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.307191    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-data\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.340564 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.340542    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istiod-ca-cert\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.340700 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.307211    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-credential-socket\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.359480 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.358818    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-envoy\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.363551 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.362981    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-podinfo\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.397517 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.395151    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-token\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.397517 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.395826    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr9m9\" (UniqueName: \"kubernetes.io/projected/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-kube-api-access-rr9m9\") pod \"e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") " pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.523531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.522754    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:58.596549 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.596510    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl"]
Oct 31 01:50:58.599373 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:58.599340    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww"]
Oct 31 01:50:58.641802 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:50:58.639978    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:50:59.002470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.000610    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"]
Oct 31 01:50:59.066717 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:50:59.066668    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d96b2a_ab79_4c0a_a187_766ef1a58c0f.slice/crio-176558964bb185aa1b3badd4cda7ba88213ea7f84a1f911fcbbbbb5508c61dcb WatchSource:0}: Error finding container 176558964bb185aa1b3badd4cda7ba88213ea7f84a1f911fcbbbbb5508c61dcb: Status 404 returned error can't find the container with id 176558964bb185aa1b3badd4cda7ba88213ea7f84a1f911fcbbbbb5508c61dcb
Oct 31 01:50:59.075586 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.075516    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:50:59.075766 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.075651    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:50:59.075766 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.075699    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:50:59.150107 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.150064    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"]
Oct 31 01:50:59.163584 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.158139    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"]
Oct 31 01:50:59.163584 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.158258    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.166992 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.166962    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-c06ae4-openshift-default-dockercfg-qjsdr\""
Oct 31 01:50:59.167833 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.167804    5650 patch_prober.go:29] interesting pod/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 31 01:50:59.168039 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.168013    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn" podUID="dcbe8798-c240-4556-bc46-d28d7ffacef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 31 01:50:59.183813 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.182830    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-credential-socket\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.184316 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.184109    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8pdh\" (UniqueName: \"kubernetes.io/projected/fec61f6a-7103-49ef-ae2c-bbe826dcd159-kube-api-access-z8pdh\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.184867 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.184844    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istiod-ca-cert\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.185456 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.185419    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-workload-socket\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.185613 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.185598    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-workload-certs\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.185752 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.185735    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-data\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.185867 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.185854    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-envoy\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.186008 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.185993    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-token\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.186122 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.186105    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-podinfo\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.280269 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.280232    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr" event={"ID":"04d96b2a-ab79-4c0a-a187-766ef1a58c0f","Type":"ContainerStarted","Data":"7c31e35ad7c443b677c136f4d46b61312e5fabee20541ea266b9eb982ada8732"}
Oct 31 01:50:59.280440 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.280286    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr" event={"ID":"04d96b2a-ab79-4c0a-a187-766ef1a58c0f","Type":"ContainerStarted","Data":"176558964bb185aa1b3badd4cda7ba88213ea7f84a1f911fcbbbbb5508c61dcb"}
Oct 31 01:50:59.290108 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.287700    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-credential-socket\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.290108 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.287778    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8pdh\" (UniqueName: \"kubernetes.io/projected/fec61f6a-7103-49ef-ae2c-bbe826dcd159-kube-api-access-z8pdh\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.290108 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.287815    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istiod-ca-cert\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.290108 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.287874    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-workload-socket\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.290108 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.287904    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-workload-certs\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.290108 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.287958    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-data\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.290108 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.288001    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-envoy\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.290108 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.288051    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-token\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.290108 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.288080    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-podinfo\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.290108 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.288161    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-credential-socket\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.290108 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.288643    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-workload-certs\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.290108 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.289715    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istiod-ca-cert\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.290108 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.289854    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-data\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.290108 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.290027    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-workload-socket\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.292156 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.292106    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-podinfo\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.294189 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.294120    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-envoy\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.411253 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.411151    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-token\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.427332 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.427235    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8pdh\" (UniqueName: \"kubernetes.io/projected/fec61f6a-7103-49ef-ae2c-bbe826dcd159-kube-api-access-z8pdh\") pod \"e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") " pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:50:59.438954 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.438877    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr" podStartSLOduration=1.438839087 podStartE2EDuration="1.438839087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:50:59.377524865 +0000 UTC m=+1815.381239158" watchObservedRunningTime="2026-10-31 01:50:59.438839087 +0000 UTC m=+1815.442553381"
Oct 31 01:50:59.546572 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.546482    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:50:59.554076 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:50:59.553764    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:51:00.082970 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:51:00.082930    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec61f6a_7103_49ef_ae2c_bbe826dcd159.slice/crio-f471e525229cfcf6af1e5c3b1a80d7e1ccb3f3f66adb39705b6fb34211c57876 WatchSource:0}: Error finding container f471e525229cfcf6af1e5c3b1a80d7e1ccb3f3f66adb39705b6fb34211c57876: Status 404 returned error can't find the container with id f471e525229cfcf6af1e5c3b1a80d7e1ccb3f3f66adb39705b6fb34211c57876
Oct 31 01:51:00.085322 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:00.085297    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"]
Oct 31 01:51:00.085674 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:00.085642    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:51:00.085732 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:00.085706    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:51:00.085768 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:00.085735    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:51:00.164364 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:00.164337    5650 patch_prober.go:29] interesting pod/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 31 01:51:00.164513 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:00.164382    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn" podUID="dcbe8798-c240-4556-bc46-d28d7ffacef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 31 01:51:00.308696 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:00.307734    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t" event={"ID":"fec61f6a-7103-49ef-ae2c-bbe826dcd159","Type":"ContainerStarted","Data":"b279e1bd517d054265149f6052141ba7260ab6f000c754a18f21c3aa903f9f2c"}
Oct 31 01:51:00.308696 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:00.307793    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t" event={"ID":"fec61f6a-7103-49ef-ae2c-bbe826dcd159","Type":"ContainerStarted","Data":"f471e525229cfcf6af1e5c3b1a80d7e1ccb3f3f66adb39705b6fb34211c57876"}
Oct 31 01:51:00.345698 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:00.345611    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t" podStartSLOduration=1.345590447 podStartE2EDuration="1.345590447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:51:00.343598884 +0000 UTC m=+1816.347313176" watchObservedRunningTime="2026-10-31 01:51:00.345590447 +0000 UTC m=+1816.349304739"
Oct 31 01:51:00.545175 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:00.545139    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:51:00.555025 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:00.554990    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:51:01.168721 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:01.168687    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:51:01.169084 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:01.169063    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:51:01.169832 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:01.169813    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:51:01.315524 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:01.315492    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:51:01.316626 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:01.316600    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:51:01.561002 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:01.560972    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:51:02.324458 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:02.324410    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:51:02.325328 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:02.325304    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:51:03.248846 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.248804    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q"]
Oct 31 01:51:03.253597 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.253562    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q"
Oct 31 01:51:03.264812 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.264780    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-726dd1-dockercfg-vbmfj\""
Oct 31 01:51:03.264957 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.264878    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-726dd1\""
Oct 31 01:51:03.268567 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.268540    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q"]
Oct 31 01:51:03.297973 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.297940    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc"]
Oct 31 01:51:03.298807 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.298774    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c759abc-5d22-4c0a-884e-5d99bcf79fdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q\" (UID: \"0c759abc-5d22-4c0a-884e-5d99bcf79fdb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q"
Oct 31 01:51:03.299128 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.299095    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq97d\" (UniqueName: \"kubernetes.io/projected/0c759abc-5d22-4c0a-884e-5d99bcf79fdb-kube-api-access-xq97d\") pod \"payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q\" (UID: \"0c759abc-5d22-4c0a-884e-5d99bcf79fdb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q"
Oct 31 01:51:03.302816 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.302792    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc"
Oct 31 01:51:03.317366 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.317327    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc"]
Oct 31 01:51:03.457475 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.445046    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c759abc-5d22-4c0a-884e-5d99bcf79fdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q\" (UID: \"0c759abc-5d22-4c0a-884e-5d99bcf79fdb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q"
Oct 31 01:51:03.457475 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.445230    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ae8229-51c6-4263-a3b2-ec6616c90d0f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc\" (UID: \"58ae8229-51c6-4263-a3b2-ec6616c90d0f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc"
Oct 31 01:51:03.457475 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.445414    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjp48\" (UniqueName: \"kubernetes.io/projected/58ae8229-51c6-4263-a3b2-ec6616c90d0f-kube-api-access-qjp48\") pod \"payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc\" (UID: \"58ae8229-51c6-4263-a3b2-ec6616c90d0f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc"
Oct 31 01:51:03.457475 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.445519    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq97d\" (UniqueName: \"kubernetes.io/projected/0c759abc-5d22-4c0a-884e-5d99bcf79fdb-kube-api-access-xq97d\") pod \"payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q\" (UID: \"0c759abc-5d22-4c0a-884e-5d99bcf79fdb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q"
Oct 31 01:51:03.503875 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.493223    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c759abc-5d22-4c0a-884e-5d99bcf79fdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q\" (UID: \"0c759abc-5d22-4c0a-884e-5d99bcf79fdb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q"
Oct 31 01:51:03.530185 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.530111    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq97d\" (UniqueName: \"kubernetes.io/projected/0c759abc-5d22-4c0a-884e-5d99bcf79fdb-kube-api-access-xq97d\") pod \"payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q\" (UID: \"0c759abc-5d22-4c0a-884e-5d99bcf79fdb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q"
Oct 31 01:51:03.573461 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.573133    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q"
Oct 31 01:51:03.605125 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.601992    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl" podUID="9cce0ae0-583a-4f7f-af47-0b5b8fde042b" containerName="payload-processing" containerID="cri-o://b20a32d2e3ba1e811ca3980dd61da593c80d447a59be8844f4acf2bd7e9c0021" gracePeriod=25
Oct 31 01:51:03.605125 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.603366    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ae8229-51c6-4263-a3b2-ec6616c90d0f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc\" (UID: \"58ae8229-51c6-4263-a3b2-ec6616c90d0f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc"
Oct 31 01:51:03.605125 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.603528    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjp48\" (UniqueName: \"kubernetes.io/projected/58ae8229-51c6-4263-a3b2-ec6616c90d0f-kube-api-access-qjp48\") pod \"payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc\" (UID: \"58ae8229-51c6-4263-a3b2-ec6616c90d0f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc"
Oct 31 01:51:03.605125 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.604651    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww" podUID="35182dd9-f44c-4d37-b9de-979018244405" containerName="payload-pre-processing" containerID="cri-o://247c273cc407bdc345fd1aef4343cc4b983600172781d18a7d554e2ca8beba1b" gracePeriod=25
Oct 31 01:51:03.605604 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.605371    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ae8229-51c6-4263-a3b2-ec6616c90d0f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc\" (UID: \"58ae8229-51c6-4263-a3b2-ec6616c90d0f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc"
Oct 31 01:51:03.628476 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.627430    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjp48\" (UniqueName: \"kubernetes.io/projected/58ae8229-51c6-4263-a3b2-ec6616c90d0f-kube-api-access-qjp48\") pod \"payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc\" (UID: \"58ae8229-51c6-4263-a3b2-ec6616c90d0f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc"
Oct 31 01:51:03.651494 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.649573    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc"
Oct 31 01:51:03.699906 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.699853    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl"]
Oct 31 01:51:03.700979 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:03.700734    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl" podUID="744d9078-58b7-4c9c-b92d-e36ff748e020" containerName="payload-pre-processing" containerID="cri-o://ff7dcec315e51adb18798e79342c018b216f75026cff0694ec4463c304f82acf" gracePeriod=30
Oct 31 01:51:04.199668 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.199637    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6"]
Oct 31 01:51:04.206956 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.206917    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-073e24-j7xg6"]
Oct 31 01:51:04.233465 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.233354    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q"]
Oct 31 01:51:04.291869 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.291836    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww"
Oct 31 01:51:04.393057 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.393008    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl" event={"ID":"9cce0ae0-583a-4f7f-af47-0b5b8fde042b","Type":"ContainerDied","Data":"b20a32d2e3ba1e811ca3980dd61da593c80d447a59be8844f4acf2bd7e9c0021"}
Oct 31 01:51:04.393666 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.392629    5650 generic.go:356] "Generic (PLEG): container finished" podID="9cce0ae0-583a-4f7f-af47-0b5b8fde042b" containerID="b20a32d2e3ba1e811ca3980dd61da593c80d447a59be8844f4acf2bd7e9c0021" exitCode=0
Oct 31 01:51:04.393994 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.393959    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl" event={"ID":"9cce0ae0-583a-4f7f-af47-0b5b8fde042b","Type":"ContainerDied","Data":"61ace603c07dd13971257ed005e962a3fa5cb97e7e7ae14f69819a45aede4605"}
Oct 31 01:51:04.394087 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.394021    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61ace603c07dd13971257ed005e962a3fa5cb97e7e7ae14f69819a45aede4605"
Oct 31 01:51:04.396606 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.396570    5650 generic.go:356] "Generic (PLEG): container finished" podID="35182dd9-f44c-4d37-b9de-979018244405" containerID="247c273cc407bdc345fd1aef4343cc4b983600172781d18a7d554e2ca8beba1b" exitCode=0
Oct 31 01:51:04.396831 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.396698    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww" event={"ID":"35182dd9-f44c-4d37-b9de-979018244405","Type":"ContainerDied","Data":"247c273cc407bdc345fd1aef4343cc4b983600172781d18a7d554e2ca8beba1b"}
Oct 31 01:51:04.397002 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.396976    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww" event={"ID":"35182dd9-f44c-4d37-b9de-979018244405","Type":"ContainerDied","Data":"d8e4663c7d84b667fafebde4a6469ec14c24b0fca66b9d32aa03f4b52853267b"}
Oct 31 01:51:04.397152 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.397131    5650 scope.go:122] "RemoveContainer" containerID="247c273cc407bdc345fd1aef4343cc4b983600172781d18a7d554e2ca8beba1b"
Oct 31 01:51:04.397580 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.396871    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww"
Oct 31 01:51:04.400723 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.400699    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q" event={"ID":"0c759abc-5d22-4c0a-884e-5d99bcf79fdb","Type":"ContainerStarted","Data":"54606dc510d5be237fb7f906c96cdbd49f73b938f4c311bdc5ff9a2ddd5fafee"}
Oct 31 01:51:04.402221 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.402194    5650 generic.go:356] "Generic (PLEG): container finished" podID="744d9078-58b7-4c9c-b92d-e36ff748e020" containerID="ff7dcec315e51adb18798e79342c018b216f75026cff0694ec4463c304f82acf" exitCode=0
Oct 31 01:51:04.402327 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.402285    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl" event={"ID":"744d9078-58b7-4c9c-b92d-e36ff748e020","Type":"ContainerDied","Data":"ff7dcec315e51adb18798e79342c018b216f75026cff0694ec4463c304f82acf"}
Oct 31 01:51:04.417415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.417394    5650 scope.go:122] "RemoveContainer" containerID="247c273cc407bdc345fd1aef4343cc4b983600172781d18a7d554e2ca8beba1b"
Oct 31 01:51:04.417844 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:51:04.417815    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"247c273cc407bdc345fd1aef4343cc4b983600172781d18a7d554e2ca8beba1b\": container with ID starting with 247c273cc407bdc345fd1aef4343cc4b983600172781d18a7d554e2ca8beba1b not found: ID does not exist" containerID="247c273cc407bdc345fd1aef4343cc4b983600172781d18a7d554e2ca8beba1b"
Oct 31 01:51:04.417905 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.417863    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"247c273cc407bdc345fd1aef4343cc4b983600172781d18a7d554e2ca8beba1b"} err="failed to get container status \"247c273cc407bdc345fd1aef4343cc4b983600172781d18a7d554e2ca8beba1b\": rpc error: code = NotFound desc = could not find container \"247c273cc407bdc345fd1aef4343cc4b983600172781d18a7d554e2ca8beba1b\": container with ID starting with 247c273cc407bdc345fd1aef4343cc4b983600172781d18a7d554e2ca8beba1b not found: ID does not exist"
Oct 31 01:51:04.422562 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.422530    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl"
Oct 31 01:51:04.425738 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.425711    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl"
Oct 31 01:51:04.448796 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.448758    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35182dd9-f44c-4d37-b9de-979018244405-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35182dd9-f44c-4d37-b9de-979018244405-plugins-config-volume\") pod \"35182dd9-f44c-4d37-b9de-979018244405\" (UID: \"35182dd9-f44c-4d37-b9de-979018244405\") "
Oct 31 01:51:04.449049 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.449023    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35182dd9-f44c-4d37-b9de-979018244405-kube-api-access-ft4r6\" (UniqueName: \"kubernetes.io/projected/35182dd9-f44c-4d37-b9de-979018244405-kube-api-access-ft4r6\") pod \"35182dd9-f44c-4d37-b9de-979018244405\" (UID: \"35182dd9-f44c-4d37-b9de-979018244405\") "
Oct 31 01:51:04.449856 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.449830    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35182dd9-f44c-4d37-b9de-979018244405-plugins-config-volume" pod "35182dd9-f44c-4d37-b9de-979018244405" (UID: "35182dd9-f44c-4d37-b9de-979018244405"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:04.451882 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.451859    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35182dd9-f44c-4d37-b9de-979018244405-kube-api-access-ft4r6" pod "35182dd9-f44c-4d37-b9de-979018244405" (UID: "35182dd9-f44c-4d37-b9de-979018244405"). InnerVolumeSpecName "kube-api-access-ft4r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:04.506005 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:51:04.505975    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ae8229_51c6_4263_a3b2_ec6616c90d0f.slice/crio-9a5bab29c89d8153c76b1217a5df29d74f5ccfab4f8ff27bab500e353a38b66f WatchSource:0}: Error finding container 9a5bab29c89d8153c76b1217a5df29d74f5ccfab4f8ff27bab500e353a38b66f: Status 404 returned error can't find the container with id 9a5bab29c89d8153c76b1217a5df29d74f5ccfab4f8ff27bab500e353a38b66f
Oct 31 01:51:04.508618 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.508387    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc"]
Oct 31 01:51:04.520601 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.520569    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs"]
Oct 31 01:51:04.521595 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.521574    5650 cpu_manager.go:414] "containerMap: removing container" podUID="744d9078-58b7-4c9c-b92d-e36ff748e020" containerName="payload-pre-processing"
Oct 31 01:51:04.521675 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.521597    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="744d9078-58b7-4c9c-b92d-e36ff748e020" containerName="payload-pre-processing"
Oct 31 01:51:04.521675 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.521617    5650 cpu_manager.go:414] "containerMap: removing container" podUID="9cce0ae0-583a-4f7f-af47-0b5b8fde042b" containerName="payload-processing"
Oct 31 01:51:04.521675 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.521624    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cce0ae0-583a-4f7f-af47-0b5b8fde042b" containerName="payload-processing"
Oct 31 01:51:04.521675 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.521647    5650 cpu_manager.go:414] "containerMap: removing container" podUID="35182dd9-f44c-4d37-b9de-979018244405" containerName="payload-pre-processing"
Oct 31 01:51:04.521675 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.521654    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35182dd9-f44c-4d37-b9de-979018244405" containerName="payload-pre-processing"
Oct 31 01:51:04.521882 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.521862    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="744d9078-58b7-4c9c-b92d-e36ff748e020" containerName="payload-pre-processing"
Oct 31 01:51:04.521882 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.521879    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cce0ae0-583a-4f7f-af47-0b5b8fde042b" containerName="payload-processing"
Oct 31 01:51:04.522055 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.521894    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="35182dd9-f44c-4d37-b9de-979018244405" containerName="payload-pre-processing"
Oct 31 01:51:04.527779 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.527750    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7"]
Oct 31 01:51:04.529126 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.529094    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs"
Oct 31 01:51:04.533347 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.533326    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7"
Oct 31 01:51:04.544754 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.544726    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-c06ae4\""
Oct 31 01:51:04.550836 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.550807    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cce0ae0-583a-4f7f-af47-0b5b8fde042b-kube-api-access-vdhxr\" (UniqueName: \"kubernetes.io/projected/9cce0ae0-583a-4f7f-af47-0b5b8fde042b-kube-api-access-vdhxr\") pod \"9cce0ae0-583a-4f7f-af47-0b5b8fde042b\" (UID: \"9cce0ae0-583a-4f7f-af47-0b5b8fde042b\") "
Oct 31 01:51:04.551067 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.551044    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cce0ae0-583a-4f7f-af47-0b5b8fde042b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cce0ae0-583a-4f7f-af47-0b5b8fde042b-plugins-config-volume\") pod \"9cce0ae0-583a-4f7f-af47-0b5b8fde042b\" (UID: \"9cce0ae0-583a-4f7f-af47-0b5b8fde042b\") "
Oct 31 01:51:04.551246 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.551226    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/744d9078-58b7-4c9c-b92d-e36ff748e020-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744d9078-58b7-4c9c-b92d-e36ff748e020-plugins-config-volume\") pod \"744d9078-58b7-4c9c-b92d-e36ff748e020\" (UID: \"744d9078-58b7-4c9c-b92d-e36ff748e020\") "
Oct 31 01:51:04.551510 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.551489    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/744d9078-58b7-4c9c-b92d-e36ff748e020-kube-api-access-f9zz9\" (UniqueName: \"kubernetes.io/projected/744d9078-58b7-4c9c-b92d-e36ff748e020-kube-api-access-f9zz9\") pod \"744d9078-58b7-4c9c-b92d-e36ff748e020\" (UID: \"744d9078-58b7-4c9c-b92d-e36ff748e020\") "
Oct 31 01:51:04.552558 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.552536    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft4r6\" (UniqueName: \"kubernetes.io/projected/35182dd9-f44c-4d37-b9de-979018244405-kube-api-access-ft4r6\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:04.552959 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.552941    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35182dd9-f44c-4d37-b9de-979018244405-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:04.553095 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.553068    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744d9078-58b7-4c9c-b92d-e36ff748e020-plugins-config-volume" pod "744d9078-58b7-4c9c-b92d-e36ff748e020" (UID: "744d9078-58b7-4c9c-b92d-e36ff748e020"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:04.553560 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.553532    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cce0ae0-583a-4f7f-af47-0b5b8fde042b-plugins-config-volume" pod "9cce0ae0-583a-4f7f-af47-0b5b8fde042b" (UID: "9cce0ae0-583a-4f7f-af47-0b5b8fde042b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:04.556794 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.556764    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7"]
Oct 31 01:51:04.561132 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.561094    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cce0ae0-583a-4f7f-af47-0b5b8fde042b-kube-api-access-vdhxr" pod "9cce0ae0-583a-4f7f-af47-0b5b8fde042b" (UID: "9cce0ae0-583a-4f7f-af47-0b5b8fde042b"). InnerVolumeSpecName "kube-api-access-vdhxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:04.562114 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.562090    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-c06ae4-dockercfg-jrkc8\""
Oct 31 01:51:04.562646 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.562625    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs"]
Oct 31 01:51:04.562911 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.562885    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744d9078-58b7-4c9c-b92d-e36ff748e020-kube-api-access-f9zz9" pod "744d9078-58b7-4c9c-b92d-e36ff748e020" (UID: "744d9078-58b7-4c9c-b92d-e36ff748e020"). InnerVolumeSpecName "kube-api-access-f9zz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:04.673718 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.670793    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5qrh\" (UniqueName: \"kubernetes.io/projected/c0ab4857-a9ba-4524-9025-d2906539f9c3-kube-api-access-r5qrh\") pod \"payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7\" (UID: \"c0ab4857-a9ba-4524-9025-d2906539f9c3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7"
Oct 31 01:51:04.673718 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.670856    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ab4857-a9ba-4524-9025-d2906539f9c3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7\" (UID: \"c0ab4857-a9ba-4524-9025-d2906539f9c3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7"
Oct 31 01:51:04.673718 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.671135    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nksr\" (UniqueName: \"kubernetes.io/projected/14a414fd-b21c-4086-bae8-090b9c22ba38-kube-api-access-4nksr\") pod \"payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs\" (UID: \"14a414fd-b21c-4086-bae8-090b9c22ba38\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs"
Oct 31 01:51:04.673718 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.671184    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a414fd-b21c-4086-bae8-090b9c22ba38-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs\" (UID: \"14a414fd-b21c-4086-bae8-090b9c22ba38\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs"
Oct 31 01:51:04.673718 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.671492    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdhxr\" (UniqueName: \"kubernetes.io/projected/9cce0ae0-583a-4f7f-af47-0b5b8fde042b-kube-api-access-vdhxr\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:04.673718 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.671519    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cce0ae0-583a-4f7f-af47-0b5b8fde042b-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:04.673718 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.671533    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744d9078-58b7-4c9c-b92d-e36ff748e020-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:04.673718 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.671546    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9zz9\" (UniqueName: \"kubernetes.io/projected/744d9078-58b7-4c9c-b92d-e36ff748e020-kube-api-access-f9zz9\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:04.759258 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.758950    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww"]
Oct 31 01:51:04.765601 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.765570    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e52ba1-f76cdc67f-2bgww"]
Oct 31 01:51:04.772392 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.772362    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nksr\" (UniqueName: \"kubernetes.io/projected/14a414fd-b21c-4086-bae8-090b9c22ba38-kube-api-access-4nksr\") pod \"payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs\" (UID: \"14a414fd-b21c-4086-bae8-090b9c22ba38\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs"
Oct 31 01:51:04.772607 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.772413    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a414fd-b21c-4086-bae8-090b9c22ba38-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs\" (UID: \"14a414fd-b21c-4086-bae8-090b9c22ba38\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs"
Oct 31 01:51:04.772607 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.772594    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5qrh\" (UniqueName: \"kubernetes.io/projected/c0ab4857-a9ba-4524-9025-d2906539f9c3-kube-api-access-r5qrh\") pod \"payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7\" (UID: \"c0ab4857-a9ba-4524-9025-d2906539f9c3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7"
Oct 31 01:51:04.772729 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.772639    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ab4857-a9ba-4524-9025-d2906539f9c3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7\" (UID: \"c0ab4857-a9ba-4524-9025-d2906539f9c3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7"
Oct 31 01:51:04.773687 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.773637    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ab4857-a9ba-4524-9025-d2906539f9c3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7\" (UID: \"c0ab4857-a9ba-4524-9025-d2906539f9c3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7"
Oct 31 01:51:04.774195 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.774151    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a414fd-b21c-4086-bae8-090b9c22ba38-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs\" (UID: \"14a414fd-b21c-4086-bae8-090b9c22ba38\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs"
Oct 31 01:51:04.794246 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.794212    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nksr\" (UniqueName: \"kubernetes.io/projected/14a414fd-b21c-4086-bae8-090b9c22ba38-kube-api-access-4nksr\") pod \"payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs\" (UID: \"14a414fd-b21c-4086-bae8-090b9c22ba38\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs"
Oct 31 01:51:04.795133 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:04.795089    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5qrh\" (UniqueName: \"kubernetes.io/projected/c0ab4857-a9ba-4524-9025-d2906539f9c3-kube-api-access-r5qrh\") pod \"payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7\" (UID: \"c0ab4857-a9ba-4524-9025-d2906539f9c3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7"
Oct 31 01:51:05.032279 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:05.031979    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs"
Oct 31 01:51:05.040732 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:05.040698    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7"
Oct 31 01:51:05.363206 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:51:05.363171    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ab4857_a9ba_4524_9025_d2906539f9c3.slice/crio-41fcaf41650b4e61b7169c08ec217f1fa363820ba463453c146fb45a5d405568 WatchSource:0}: Error finding container 41fcaf41650b4e61b7169c08ec217f1fa363820ba463453c146fb45a5d405568: Status 404 returned error can't find the container with id 41fcaf41650b4e61b7169c08ec217f1fa363820ba463453c146fb45a5d405568
Oct 31 01:51:05.364065 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:51:05.364019    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a414fd_b21c_4086_bae8_090b9c22ba38.slice/crio-c41c93136ebcc4baac7a328991a7a9d39732220ea23bb80e100af05ecbd93060 WatchSource:0}: Error finding container c41c93136ebcc4baac7a328991a7a9d39732220ea23bb80e100af05ecbd93060: Status 404 returned error can't find the container with id c41c93136ebcc4baac7a328991a7a9d39732220ea23bb80e100af05ecbd93060
Oct 31 01:51:05.364884 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:05.364556    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7"]
Oct 31 01:51:05.369141 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:05.369119    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs"]
Oct 31 01:51:05.414122 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:05.414089    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q" event={"ID":"0c759abc-5d22-4c0a-884e-5d99bcf79fdb","Type":"ContainerStarted","Data":"091fc9254cd3926924319c8a44fce8cb689d5b08155c1b5ad8d0ba5fb59c0d66"}
Oct 31 01:51:05.414362 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:05.414337    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q"
Oct 31 01:51:05.416146 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:05.416114    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc" event={"ID":"58ae8229-51c6-4263-a3b2-ec6616c90d0f","Type":"ContainerStarted","Data":"9bdc3cbb1a0de66f2b6036dc4ac64920a2853329964824fb0814a1a7348c310d"}
Oct 31 01:51:05.416265 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:05.416169    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc" event={"ID":"58ae8229-51c6-4263-a3b2-ec6616c90d0f","Type":"ContainerStarted","Data":"9a5bab29c89d8153c76b1217a5df29d74f5ccfab4f8ff27bab500e353a38b66f"}
Oct 31 01:51:05.416265 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:05.416218    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc"
Oct 31 01:51:05.417664 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:05.417640    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl"
Oct 31 01:51:05.417770 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:05.417658    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl" event={"ID":"744d9078-58b7-4c9c-b92d-e36ff748e020","Type":"ContainerDied","Data":"9c342d69dc06b4b809ab4b705292c3bc3bcd60ee001f57d952c47bec1bc44ed2"}
Oct 31 01:51:05.417770 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:05.417710    5650 scope.go:122] "RemoveContainer" containerID="ff7dcec315e51adb18798e79342c018b216f75026cff0694ec4463c304f82acf"
Oct 31 01:51:05.419780 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:05.419758    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7" event={"ID":"c0ab4857-a9ba-4524-9025-d2906539f9c3","Type":"ContainerStarted","Data":"41fcaf41650b4e61b7169c08ec217f1fa363820ba463453c146fb45a5d405568"}
Oct 31 01:51:05.421732 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:05.421706    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs" event={"ID":"14a414fd-b21c-4086-bae8-090b9c22ba38","Type":"ContainerStarted","Data":"c41c93136ebcc4baac7a328991a7a9d39732220ea23bb80e100af05ecbd93060"}
Oct 31 01:51:05.421732 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:05.421728    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl"
Oct 31 01:51:05.439395 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:05.439347    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q" podStartSLOduration=2.439333993 podStartE2EDuration="2.439333993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:51:05.435276118 +0000 UTC m=+1821.438990417" watchObservedRunningTime="2026-10-31 01:51:05.439333993 +0000 UTC m=+1821.443048298"
Oct 31 01:51:05.470132 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:05.470061    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl"]
Oct 31 01:51:05.493496 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:05.490492    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-073e24-96b5fbc75-nrqpl"]
Oct 31 01:51:05.494859 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:05.494788    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc" podStartSLOduration=2.494769872 podStartE2EDuration="2.494769872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:51:05.482003275 +0000 UTC m=+1821.485717568" watchObservedRunningTime="2026-10-31 01:51:05.494769872 +0000 UTC m=+1821.498484164"
Oct 31 01:51:05.506013 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:05.505969    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl"]
Oct 31 01:51:05.511464 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:05.511423    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e52ba1-6ff49f9b65-fkztl"]
Oct 31 01:51:05.990877 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:05.990840    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"]
Oct 31 01:51:05.991178 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:05.991088    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql" podUID="0bcd9014-81f9-45a5-a32f-8cbe21750e80" containerName="istio-proxy" containerID="cri-o://ae5731e27b45330d7d678aca88b17475a6a2822c54aa67487dfa9c6e976647bb" gracePeriod=30
Oct 31 01:51:06.006463 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:06.006418    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"]
Oct 31 01:51:06.007490 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:06.007319    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5" podUID="07620cf8-e211-41cf-8faf-177eeb1e2cac" containerName="istio-proxy" containerID="cri-o://acc8ac22b724b78683178db63b2fdb0e24da950588fbaade22e7cbd24cbf9846" gracePeriod=30
Oct 31 01:51:06.120243 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:06.120209    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f4da93e-cf46-4af6-9255-1c1e150eb67c" path="/var/lib/kubelet/pods/0f4da93e-cf46-4af6-9255-1c1e150eb67c/volumes"
Oct 31 01:51:06.120755 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:06.120740    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35182dd9-f44c-4d37-b9de-979018244405" path="/var/lib/kubelet/pods/35182dd9-f44c-4d37-b9de-979018244405/volumes"
Oct 31 01:51:06.121144 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:06.121130    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="744d9078-58b7-4c9c-b92d-e36ff748e020" path="/var/lib/kubelet/pods/744d9078-58b7-4c9c-b92d-e36ff748e020/volumes"
Oct 31 01:51:06.121916 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:06.121900    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cce0ae0-583a-4f7f-af47-0b5b8fde042b" path="/var/lib/kubelet/pods/9cce0ae0-583a-4f7f-af47-0b5b8fde042b/volumes"
Oct 31 01:51:06.431029 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:06.430925    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7" event={"ID":"c0ab4857-a9ba-4524-9025-d2906539f9c3","Type":"ContainerStarted","Data":"7219246d9bb388351b0d5d7cdb622128a1c1b680cf3ac00f0551b6bace33a7ad"}
Oct 31 01:51:06.431029 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:06.431006    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7"
Oct 31 01:51:06.432777 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:06.432742    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs" event={"ID":"14a414fd-b21c-4086-bae8-090b9c22ba38","Type":"ContainerStarted","Data":"747f8d17c5dc8f293b5277bebae773ffc92deb039338fbf9bc5ae6fb1d588962"}
Oct 31 01:51:06.432909 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:06.432831    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs"
Oct 31 01:51:06.464606 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:06.463605    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7" podStartSLOduration=2.463582559 podStartE2EDuration="2.463582559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:51:06.454697979 +0000 UTC m=+1822.458412273" watchObservedRunningTime="2026-10-31 01:51:06.463582559 +0000 UTC m=+1822.467296855"
Oct 31 01:51:06.482136 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:06.482064    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs" podStartSLOduration=2.482044692 podStartE2EDuration="2.482044692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:51:06.478962327 +0000 UTC m=+1822.482676619" watchObservedRunningTime="2026-10-31 01:51:06.482044692 +0000 UTC m=+1822.485758986"
Oct 31 01:51:11.371114 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.371082    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:51:11.491774 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.491715    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-podinfo\") pod \"07620cf8-e211-41cf-8faf-177eeb1e2cac\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") "
Oct 31 01:51:11.491985 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.491797    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-credential-socket\") pod \"07620cf8-e211-41cf-8faf-177eeb1e2cac\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") "
Oct 31 01:51:11.491985 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.491842    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-workload-certs\") pod \"07620cf8-e211-41cf-8faf-177eeb1e2cac\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") "
Oct 31 01:51:11.491985 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.491902    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-data\") pod \"07620cf8-e211-41cf-8faf-177eeb1e2cac\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") "
Oct 31 01:51:11.491985 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.491953    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07620cf8-e211-41cf-8faf-177eeb1e2cac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07620cf8-e211-41cf-8faf-177eeb1e2cac-istiod-ca-cert\") pod \"07620cf8-e211-41cf-8faf-177eeb1e2cac\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") "
Oct 31 01:51:11.492201 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.492096    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07620cf8-e211-41cf-8faf-177eeb1e2cac-kube-api-access-xdfkk\" (UniqueName: \"kubernetes.io/projected/07620cf8-e211-41cf-8faf-177eeb1e2cac-kube-api-access-xdfkk\") pod \"07620cf8-e211-41cf-8faf-177eeb1e2cac\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") "
Oct 31 01:51:11.492201 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.492118    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-credential-socket" pod "07620cf8-e211-41cf-8faf-177eeb1e2cac" (UID: "07620cf8-e211-41cf-8faf-177eeb1e2cac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:11.492201 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.492130    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-workload-socket\") pod \"07620cf8-e211-41cf-8faf-177eeb1e2cac\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") "
Oct 31 01:51:11.492364 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.492243    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-token\" (UniqueName: \"kubernetes.io/projected/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-token\") pod \"07620cf8-e211-41cf-8faf-177eeb1e2cac\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") "
Oct 31 01:51:11.492364 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.492244    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-workload-certs" pod "07620cf8-e211-41cf-8faf-177eeb1e2cac" (UID: "07620cf8-e211-41cf-8faf-177eeb1e2cac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:11.492364 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.492252    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-data" pod "07620cf8-e211-41cf-8faf-177eeb1e2cac" (UID: "07620cf8-e211-41cf-8faf-177eeb1e2cac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:11.492551 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.492390    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-envoy\") pod \"07620cf8-e211-41cf-8faf-177eeb1e2cac\" (UID: \"07620cf8-e211-41cf-8faf-177eeb1e2cac\") "
Oct 31 01:51:11.492551 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.492402    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-workload-socket" pod "07620cf8-e211-41cf-8faf-177eeb1e2cac" (UID: "07620cf8-e211-41cf-8faf-177eeb1e2cac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:11.492551 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.492425    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07620cf8-e211-41cf-8faf-177eeb1e2cac-istiod-ca-cert" pod "07620cf8-e211-41cf-8faf-177eeb1e2cac" (UID: "07620cf8-e211-41cf-8faf-177eeb1e2cac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:11.493086 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.493062    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:11.493086 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.493085    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:11.493269 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.493096    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:11.493269 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.493114    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07620cf8-e211-41cf-8faf-177eeb1e2cac-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:11.493269 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.493124    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:11.494087 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.494060    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-podinfo" pod "07620cf8-e211-41cf-8faf-177eeb1e2cac" (UID: "07620cf8-e211-41cf-8faf-177eeb1e2cac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:51:11.494507 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.494482    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-token" pod "07620cf8-e211-41cf-8faf-177eeb1e2cac" (UID: "07620cf8-e211-41cf-8faf-177eeb1e2cac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:11.494718 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.494695    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-envoy" pod "07620cf8-e211-41cf-8faf-177eeb1e2cac" (UID: "07620cf8-e211-41cf-8faf-177eeb1e2cac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:11.494793 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.494745    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07620cf8-e211-41cf-8faf-177eeb1e2cac-kube-api-access-xdfkk" pod "07620cf8-e211-41cf-8faf-177eeb1e2cac" (UID: "07620cf8-e211-41cf-8faf-177eeb1e2cac"). InnerVolumeSpecName "kube-api-access-xdfkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:11.535703 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.535670    5650 generic.go:356] "Generic (PLEG): container finished" podID="0bcd9014-81f9-45a5-a32f-8cbe21750e80" containerID="ae5731e27b45330d7d678aca88b17475a6a2822c54aa67487dfa9c6e976647bb" exitCode=0
Oct 31 01:51:11.535872 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.535748    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql" event={"ID":"0bcd9014-81f9-45a5-a32f-8cbe21750e80","Type":"ContainerDied","Data":"ae5731e27b45330d7d678aca88b17475a6a2822c54aa67487dfa9c6e976647bb"}
Oct 31 01:51:11.537256 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.537230    5650 generic.go:356] "Generic (PLEG): container finished" podID="07620cf8-e211-41cf-8faf-177eeb1e2cac" containerID="acc8ac22b724b78683178db63b2fdb0e24da950588fbaade22e7cbd24cbf9846" exitCode=0
Oct 31 01:51:11.537369 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.537281    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"
Oct 31 01:51:11.537369 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.537307    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5" event={"ID":"07620cf8-e211-41cf-8faf-177eeb1e2cac","Type":"ContainerDied","Data":"acc8ac22b724b78683178db63b2fdb0e24da950588fbaade22e7cbd24cbf9846"}
Oct 31 01:51:11.537369 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.537357    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5" event={"ID":"07620cf8-e211-41cf-8faf-177eeb1e2cac","Type":"ContainerDied","Data":"af51d592371d9b8d7ad52d6d2e23363caa00b61a02f6635ffcff6a19ca0cd33d"}
Oct 31 01:51:11.537492 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.537383    5650 scope.go:122] "RemoveContainer" containerID="acc8ac22b724b78683178db63b2fdb0e24da950588fbaade22e7cbd24cbf9846"
Oct 31 01:51:11.550934 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.550913    5650 scope.go:122] "RemoveContainer" containerID="acc8ac22b724b78683178db63b2fdb0e24da950588fbaade22e7cbd24cbf9846"
Oct 31 01:51:11.551284 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:51:11.551265    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acc8ac22b724b78683178db63b2fdb0e24da950588fbaade22e7cbd24cbf9846\": container with ID starting with acc8ac22b724b78683178db63b2fdb0e24da950588fbaade22e7cbd24cbf9846 not found: ID does not exist" containerID="acc8ac22b724b78683178db63b2fdb0e24da950588fbaade22e7cbd24cbf9846"
Oct 31 01:51:11.551337 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.551296    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc8ac22b724b78683178db63b2fdb0e24da950588fbaade22e7cbd24cbf9846"} err="failed to get container status \"acc8ac22b724b78683178db63b2fdb0e24da950588fbaade22e7cbd24cbf9846\": rpc error: code = NotFound desc = could not find container \"acc8ac22b724b78683178db63b2fdb0e24da950588fbaade22e7cbd24cbf9846\": container with ID starting with acc8ac22b724b78683178db63b2fdb0e24da950588fbaade22e7cbd24cbf9846 not found: ID does not exist"
Oct 31 01:51:11.569376 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.569337    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"]
Oct 31 01:51:11.575584 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.575558    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-wpqq5"]
Oct 31 01:51:11.598341 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.598220    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:11.598341 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.598280    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:11.598341 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.598299    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdfkk\" (UniqueName: \"kubernetes.io/projected/07620cf8-e211-41cf-8faf-177eeb1e2cac-kube-api-access-xdfkk\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:11.598341 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.598316    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07620cf8-e211-41cf-8faf-177eeb1e2cac-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:11.656927 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.656899    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:51:11.699893 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.699854    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-workload-certs\") pod \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") "
Oct 31 01:51:11.700088 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.699988    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-podinfo\") pod \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") "
Oct 31 01:51:11.700088 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.700019    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-token\" (UniqueName: \"kubernetes.io/projected/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-token\") pod \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") "
Oct 31 01:51:11.700207 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.700103    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-workload-socket\") pod \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") "
Oct 31 01:51:11.700207 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.700150    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bcd9014-81f9-45a5-a32f-8cbe21750e80-kube-api-access-2d8cg\" (UniqueName: \"kubernetes.io/projected/0bcd9014-81f9-45a5-a32f-8cbe21750e80-kube-api-access-2d8cg\") pod \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") "
Oct 31 01:51:11.700207 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.700172    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-envoy\") pod \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") "
Oct 31 01:51:11.700207 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.700173    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-workload-certs" pod "0bcd9014-81f9-45a5-a32f-8cbe21750e80" (UID: "0bcd9014-81f9-45a5-a32f-8cbe21750e80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:11.700483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.700278    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istiod-ca-cert\") pod \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") "
Oct 31 01:51:11.700483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.700298    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-credential-socket\") pod \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") "
Oct 31 01:51:11.700483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.700384    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-data\") pod \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\" (UID: \"0bcd9014-81f9-45a5-a32f-8cbe21750e80\") "
Oct 31 01:51:11.701084 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.701056    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:11.701548 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.701527    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-data" pod "0bcd9014-81f9-45a5-a32f-8cbe21750e80" (UID: "0bcd9014-81f9-45a5-a32f-8cbe21750e80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:11.702134 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.702082    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-workload-socket" pod "0bcd9014-81f9-45a5-a32f-8cbe21750e80" (UID: "0bcd9014-81f9-45a5-a32f-8cbe21750e80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:11.702225 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.702157    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-credential-socket" pod "0bcd9014-81f9-45a5-a32f-8cbe21750e80" (UID: "0bcd9014-81f9-45a5-a32f-8cbe21750e80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:11.702312 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.702121    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istiod-ca-cert" pod "0bcd9014-81f9-45a5-a32f-8cbe21750e80" (UID: "0bcd9014-81f9-45a5-a32f-8cbe21750e80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:11.703180 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.703147    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-token" pod "0bcd9014-81f9-45a5-a32f-8cbe21750e80" (UID: "0bcd9014-81f9-45a5-a32f-8cbe21750e80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:11.703264 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.703160    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-envoy" pod "0bcd9014-81f9-45a5-a32f-8cbe21750e80" (UID: "0bcd9014-81f9-45a5-a32f-8cbe21750e80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:11.704287 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.704264    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-podinfo" pod "0bcd9014-81f9-45a5-a32f-8cbe21750e80" (UID: "0bcd9014-81f9-45a5-a32f-8cbe21750e80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:51:11.710457 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.710408    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bcd9014-81f9-45a5-a32f-8cbe21750e80-kube-api-access-2d8cg" pod "0bcd9014-81f9-45a5-a32f-8cbe21750e80" (UID: "0bcd9014-81f9-45a5-a32f-8cbe21750e80"). InnerVolumeSpecName "kube-api-access-2d8cg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:11.802428 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.802375    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:11.802428 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.802424    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:11.802428 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.802435    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:11.802734 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.802533    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:11.802734 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.802547    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:11.802734 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.802556    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:11.802734 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.802566    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d8cg\" (UniqueName: \"kubernetes.io/projected/0bcd9014-81f9-45a5-a32f-8cbe21750e80-kube-api-access-2d8cg\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:11.802734 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:11.802577    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bcd9014-81f9-45a5-a32f-8cbe21750e80-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:12.135076 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:12.135037    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07620cf8-e211-41cf-8faf-177eeb1e2cac" path="/var/lib/kubelet/pods/07620cf8-e211-41cf-8faf-177eeb1e2cac/volumes"
Oct 31 01:51:12.546119 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:12.546086    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"
Oct 31 01:51:12.546119 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:12.546095    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql" event={"ID":"0bcd9014-81f9-45a5-a32f-8cbe21750e80","Type":"ContainerDied","Data":"347527c774a5551380d514cee55b72e1b74a4ef6d3e154f35b8dc21cc853c8d7"}
Oct 31 01:51:12.546604 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:12.546163    5650 scope.go:122] "RemoveContainer" containerID="ae5731e27b45330d7d678aca88b17475a6a2822c54aa67487dfa9c6e976647bb"
Oct 31 01:51:12.580018 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:12.579990    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"]
Oct 31 01:51:12.587928 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:12.587903    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-j48ql"]
Oct 31 01:51:12.883820 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:12.883722    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp"]
Oct 31 01:51:12.884903 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:12.884874    5650 cpu_manager.go:414] "containerMap: removing container" podUID="0bcd9014-81f9-45a5-a32f-8cbe21750e80" containerName="istio-proxy"
Oct 31 01:51:12.885089 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:12.885070    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bcd9014-81f9-45a5-a32f-8cbe21750e80" containerName="istio-proxy"
Oct 31 01:51:12.885219 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:12.885204    5650 cpu_manager.go:414] "containerMap: removing container" podUID="07620cf8-e211-41cf-8faf-177eeb1e2cac" containerName="istio-proxy"
Oct 31 01:51:12.885301 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:12.885290    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07620cf8-e211-41cf-8faf-177eeb1e2cac" containerName="istio-proxy"
Oct 31 01:51:12.885607 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:12.885587    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bcd9014-81f9-45a5-a32f-8cbe21750e80" containerName="istio-proxy"
Oct 31 01:51:12.885724 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:12.885705    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="07620cf8-e211-41cf-8faf-177eeb1e2cac" containerName="istio-proxy"
Oct 31 01:51:12.894259 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:12.894224    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp"
Oct 31 01:51:12.901318 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:12.901290    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp"]
Oct 31 01:51:12.903408 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:12.903382    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-c06ae4-serving-cert\""
Oct 31 01:51:13.031804 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:13.031759    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2342b77-070b-4cbb-9856-f0f075c05719-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp\" (UID: \"b2342b77-070b-4cbb-9856-f0f075c05719\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp"
Oct 31 01:51:13.031976 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:13.031815    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8gqt\" (UniqueName: \"kubernetes.io/projected/b2342b77-070b-4cbb-9856-f0f075c05719-kube-api-access-c8gqt\") pod \"maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp\" (UID: \"b2342b77-070b-4cbb-9856-f0f075c05719\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp"
Oct 31 01:51:13.032147 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:13.032127    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2342b77-070b-4cbb-9856-f0f075c05719-metrics-certs\") pod \"maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp\" (UID: \"b2342b77-070b-4cbb-9856-f0f075c05719\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp"
Oct 31 01:51:13.032240 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:13.032227    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2342b77-070b-4cbb-9856-f0f075c05719-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp\" (UID: \"b2342b77-070b-4cbb-9856-f0f075c05719\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp"
Oct 31 01:51:13.134163 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:13.134082    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2342b77-070b-4cbb-9856-f0f075c05719-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp\" (UID: \"b2342b77-070b-4cbb-9856-f0f075c05719\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp"
Oct 31 01:51:13.134313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:13.134188    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2342b77-070b-4cbb-9856-f0f075c05719-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp\" (UID: \"b2342b77-070b-4cbb-9856-f0f075c05719\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp"
Oct 31 01:51:13.134313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:13.134212    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8gqt\" (UniqueName: \"kubernetes.io/projected/b2342b77-070b-4cbb-9856-f0f075c05719-kube-api-access-c8gqt\") pod \"maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp\" (UID: \"b2342b77-070b-4cbb-9856-f0f075c05719\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp"
Oct 31 01:51:13.134402 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:13.134377    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2342b77-070b-4cbb-9856-f0f075c05719-metrics-certs\") pod \"maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp\" (UID: \"b2342b77-070b-4cbb-9856-f0f075c05719\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp"
Oct 31 01:51:13.135029 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:13.135004    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2342b77-070b-4cbb-9856-f0f075c05719-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp\" (UID: \"b2342b77-070b-4cbb-9856-f0f075c05719\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp"
Oct 31 01:51:13.137068 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:13.137040    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2342b77-070b-4cbb-9856-f0f075c05719-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp\" (UID: \"b2342b77-070b-4cbb-9856-f0f075c05719\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp"
Oct 31 01:51:13.137174 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:13.137099    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2342b77-070b-4cbb-9856-f0f075c05719-metrics-certs\") pod \"maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp\" (UID: \"b2342b77-070b-4cbb-9856-f0f075c05719\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp"
Oct 31 01:51:13.148171 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:13.148126    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8gqt\" (UniqueName: \"kubernetes.io/projected/b2342b77-070b-4cbb-9856-f0f075c05719-kube-api-access-c8gqt\") pod \"maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp\" (UID: \"b2342b77-070b-4cbb-9856-f0f075c05719\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp"
Oct 31 01:51:13.219772 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:13.219737    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp"
Oct 31 01:51:13.516668 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:13.516634    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp"]
Oct 31 01:51:13.519219 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:51:13.519185    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2342b77_070b_4cbb_9856_f0f075c05719.slice/crio-1adcbb581ef6ddb12c9c6ac2d78bbd7775e4d8d639bf52afb1c93920d3261116 WatchSource:0}: Error finding container 1adcbb581ef6ddb12c9c6ac2d78bbd7775e4d8d639bf52afb1c93920d3261116: Status 404 returned error can't find the container with id 1adcbb581ef6ddb12c9c6ac2d78bbd7775e4d8d639bf52afb1c93920d3261116
Oct 31 01:51:13.521875 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:13.521836    5650 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:51:13.558063 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:13.558027    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp" event={"ID":"b2342b77-070b-4cbb-9856-f0f075c05719","Type":"ContainerStarted","Data":"1adcbb581ef6ddb12c9c6ac2d78bbd7775e4d8d639bf52afb1c93920d3261116"}
Oct 31 01:51:14.121651 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:14.121618    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bcd9014-81f9-45a5-a32f-8cbe21750e80" path="/var/lib/kubelet/pods/0bcd9014-81f9-45a5-a32f-8cbe21750e80/volumes"
Oct 31 01:51:14.155516 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:14.155486    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-lq6ms_d5f6f417-2dd1-4f4e-ad96-92d1f512cac1/payload-processing/0.log"
Oct 31 01:51:14.369352 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:14.369281    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-lq6ms_d5f6f417-2dd1-4f4e-ad96-92d1f512cac1/payload-processing/0.log"
Oct 31 01:51:14.567262 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:14.567224    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp" event={"ID":"b2342b77-070b-4cbb-9856-f0f075c05719","Type":"ContainerStarted","Data":"e7d9d94e9db481a3ce5a07ed4f503c62ea599efb1c9ac96351b258b370230cda"}
Oct 31 01:51:14.567683 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:14.567319    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp"
Oct 31 01:51:14.591496 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:14.591424    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp" podStartSLOduration=2.120149762 podStartE2EDuration="2.591409158s" totalImagesPullingTime="471.259396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:51:14.589301952 +0000 UTC m=+1830.593016266" watchObservedRunningTime="2026-10-31 01:51:14.591409158 +0000 UTC m=+1830.595123464"
Oct 31 01:51:16.437831 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:16.437803    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q"
Oct 31 01:51:16.438235 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:16.437871    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc"
Oct 31 01:51:17.454006 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:17.453973    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs"
Oct 31 01:51:17.454426 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:17.454056    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7"
Oct 31 01:51:20.582833 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:20.582803    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp"
Oct 31 01:51:26.593834 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:26.593794    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8"]
Oct 31 01:51:26.594581 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:26.594037    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8" podUID="eaf490d2-bacd-4a5b-bab2-58f20427a982" containerName="payload-processing" containerID="cri-o://7779ff426dd0f32f9fba856df891aae9b36f64c00fbe47ec8d5e93bbde417a68" gracePeriod=30
Oct 31 01:51:26.617244 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:26.617210    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt"]
Oct 31 01:51:26.617579 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:26.617516    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt" podUID="d2ace825-0835-4816-a346-7f9a72c3403e" containerName="payload-pre-processing" containerID="cri-o://480221c1bebd6bac282511539c8caf6be93bc349bbf8d0e73493d06e89c82715" gracePeriod=30
Oct 31 01:51:26.803097 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:26.796091    5650 generic.go:356] "Generic (PLEG): container finished" podID="eaf490d2-bacd-4a5b-bab2-58f20427a982" containerID="7779ff426dd0f32f9fba856df891aae9b36f64c00fbe47ec8d5e93bbde417a68" exitCode=0
Oct 31 01:51:26.803097 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:26.796144    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8" event={"ID":"eaf490d2-bacd-4a5b-bab2-58f20427a982","Type":"ContainerDied","Data":"7779ff426dd0f32f9fba856df891aae9b36f64c00fbe47ec8d5e93bbde417a68"}
Oct 31 01:51:26.803097 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:26.802643    5650 generic.go:356] "Generic (PLEG): container finished" podID="d2ace825-0835-4816-a346-7f9a72c3403e" containerID="480221c1bebd6bac282511539c8caf6be93bc349bbf8d0e73493d06e89c82715" exitCode=0
Oct 31 01:51:26.803097 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:26.802719    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt" event={"ID":"d2ace825-0835-4816-a346-7f9a72c3403e","Type":"ContainerDied","Data":"480221c1bebd6bac282511539c8caf6be93bc349bbf8d0e73493d06e89c82715"}
Oct 31 01:51:26.914331 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:26.914299    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8"
Oct 31 01:51:26.955867 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:26.955829    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt"
Oct 31 01:51:27.010179 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:27.010130    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf490d2-bacd-4a5b-bab2-58f20427a982-kube-api-access-rkmhk\" (UniqueName: \"kubernetes.io/projected/eaf490d2-bacd-4a5b-bab2-58f20427a982-kube-api-access-rkmhk\") pod \"eaf490d2-bacd-4a5b-bab2-58f20427a982\" (UID: \"eaf490d2-bacd-4a5b-bab2-58f20427a982\") "
Oct 31 01:51:27.010462 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:27.010311    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2ace825-0835-4816-a346-7f9a72c3403e-kube-api-access-bbnxk\" (UniqueName: \"kubernetes.io/projected/d2ace825-0835-4816-a346-7f9a72c3403e-kube-api-access-bbnxk\") pod \"d2ace825-0835-4816-a346-7f9a72c3403e\" (UID: \"d2ace825-0835-4816-a346-7f9a72c3403e\") "
Oct 31 01:51:27.010615 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:27.010594    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaf490d2-bacd-4a5b-bab2-58f20427a982-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaf490d2-bacd-4a5b-bab2-58f20427a982-plugins-config-volume\") pod \"eaf490d2-bacd-4a5b-bab2-58f20427a982\" (UID: \"eaf490d2-bacd-4a5b-bab2-58f20427a982\") "
Oct 31 01:51:27.010743 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:27.010633    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2ace825-0835-4816-a346-7f9a72c3403e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ace825-0835-4816-a346-7f9a72c3403e-plugins-config-volume\") pod \"d2ace825-0835-4816-a346-7f9a72c3403e\" (UID: \"d2ace825-0835-4816-a346-7f9a72c3403e\") "
Oct 31 01:51:27.012005 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:27.011849    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf490d2-bacd-4a5b-bab2-58f20427a982-plugins-config-volume" pod "eaf490d2-bacd-4a5b-bab2-58f20427a982" (UID: "eaf490d2-bacd-4a5b-bab2-58f20427a982"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:27.012005 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:27.011943    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ace825-0835-4816-a346-7f9a72c3403e-plugins-config-volume" pod "d2ace825-0835-4816-a346-7f9a72c3403e" (UID: "d2ace825-0835-4816-a346-7f9a72c3403e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:27.013336 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:27.013303    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf490d2-bacd-4a5b-bab2-58f20427a982-kube-api-access-rkmhk" pod "eaf490d2-bacd-4a5b-bab2-58f20427a982" (UID: "eaf490d2-bacd-4a5b-bab2-58f20427a982"). InnerVolumeSpecName "kube-api-access-rkmhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:27.020430 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:27.020361    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ace825-0835-4816-a346-7f9a72c3403e-kube-api-access-bbnxk" pod "d2ace825-0835-4816-a346-7f9a72c3403e" (UID: "d2ace825-0835-4816-a346-7f9a72c3403e"). InnerVolumeSpecName "kube-api-access-bbnxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:27.113759 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:27.113662    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaf490d2-bacd-4a5b-bab2-58f20427a982-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:27.113759 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:27.113699    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ace825-0835-4816-a346-7f9a72c3403e-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:27.113759 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:27.113716    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkmhk\" (UniqueName: \"kubernetes.io/projected/eaf490d2-bacd-4a5b-bab2-58f20427a982-kube-api-access-rkmhk\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:27.113759 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:27.113731    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbnxk\" (UniqueName: \"kubernetes.io/projected/d2ace825-0835-4816-a346-7f9a72c3403e-kube-api-access-bbnxk\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:27.130577 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:27.130480    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"]
Oct 31 01:51:27.133190 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:27.132549    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td" podUID="e1f59db2-d04e-4e96-bf12-5a7d3f18eb99" containerName="istio-proxy" containerID="cri-o://5fae99081b3a102413ea91fc4d1a179fd86b3e1e3516cbe35c37748cebad01e0" gracePeriod=30
Oct 31 01:51:27.812137 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:27.812107    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt"
Oct 31 01:51:27.812708 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:27.812148    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt" event={"ID":"d2ace825-0835-4816-a346-7f9a72c3403e","Type":"ContainerDied","Data":"593c0fc2ee0078f5ce7a50072daa764c438ad44c89578beea786c66620a353ef"}
Oct 31 01:51:27.812708 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:27.812210    5650 scope.go:122] "RemoveContainer" containerID="480221c1bebd6bac282511539c8caf6be93bc349bbf8d0e73493d06e89c82715"
Oct 31 01:51:27.813919 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:27.813882    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8" event={"ID":"eaf490d2-bacd-4a5b-bab2-58f20427a982","Type":"ContainerDied","Data":"7dda2e3f0bcddca218636318d10a098b805cb4a7396efd1c100bb8fec4f7bc15"}
Oct 31 01:51:27.813919 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:27.813909    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8"
Oct 31 01:51:27.829537 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:27.829511    5650 scope.go:122] "RemoveContainer" containerID="7779ff426dd0f32f9fba856df891aae9b36f64c00fbe47ec8d5e93bbde417a68"
Oct 31 01:51:27.847893 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:27.847861    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8"]
Oct 31 01:51:27.851842 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:27.851814    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90c4d1-cc54c76d6-d4bb8"]
Oct 31 01:51:27.867839 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:27.867805    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt"]
Oct 31 01:51:27.875373 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:27.875289    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-90c4d1-d575b4cd7-h8xrt"]
Oct 31 01:51:28.119698 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.119663    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2ace825-0835-4816-a346-7f9a72c3403e" path="/var/lib/kubelet/pods/d2ace825-0835-4816-a346-7f9a72c3403e/volumes"
Oct 31 01:51:28.120208 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.120191    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaf490d2-bacd-4a5b-bab2-58f20427a982" path="/var/lib/kubelet/pods/eaf490d2-bacd-4a5b-bab2-58f20427a982/volumes"
Oct 31 01:51:28.554104 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:51:28.554066    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:51:28.727177 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.727145    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"]
Oct 31 01:51:28.727980 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.727957    5650 cpu_manager.go:414] "containerMap: removing container" podUID="eaf490d2-bacd-4a5b-bab2-58f20427a982" containerName="payload-processing"
Oct 31 01:51:28.727980 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.727979    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf490d2-bacd-4a5b-bab2-58f20427a982" containerName="payload-processing"
Oct 31 01:51:28.728183 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.728043    5650 cpu_manager.go:414] "containerMap: removing container" podUID="d2ace825-0835-4816-a346-7f9a72c3403e" containerName="payload-pre-processing"
Oct 31 01:51:28.728183 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.728051    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ace825-0835-4816-a346-7f9a72c3403e" containerName="payload-pre-processing"
Oct 31 01:51:28.728288 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.728227    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf490d2-bacd-4a5b-bab2-58f20427a982" containerName="payload-processing"
Oct 31 01:51:28.728288 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.728254    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2ace825-0835-4816-a346-7f9a72c3403e" containerName="payload-pre-processing"
Oct 31 01:51:28.733770 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.733749    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.759574 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.759543    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a3ff0c-openshift-default-dockercfg-rhsrz\""
Oct 31 01:51:28.761998 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.761917    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"]
Oct 31 01:51:28.861528 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.860652    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-credential-socket\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.861528 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.860728    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-data\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.861528 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.860780    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf9p9\" (UniqueName: \"kubernetes.io/projected/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-kube-api-access-vf9p9\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.861528 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.860883    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-podinfo\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.861528 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.860956    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-workload-socket\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.861528 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.861000    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-token\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.861528 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.861028    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-envoy\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.861528 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.861064    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-workload-certs\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.861528 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.861184    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istiod-ca-cert\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.965805 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.962286    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-workload-socket\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.965805 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.962365    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-token\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.965805 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.962397    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-envoy\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.965805 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.962435    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-workload-certs\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.965805 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.962565    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istiod-ca-cert\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.965805 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.962685    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-credential-socket\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.965805 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.962736    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-data\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.965805 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.962782    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf9p9\" (UniqueName: \"kubernetes.io/projected/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-kube-api-access-vf9p9\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.965805 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.962888    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-podinfo\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.965805 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.963887    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-workload-certs\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.965805 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.964245    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-workload-socket\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.965805 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.965160    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-credential-socket\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.965805 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.965789    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istiod-ca-cert\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.974469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.970968    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-envoy\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.974469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.971791    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-data\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.974469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.972434    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-podinfo\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.982467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.978137    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-token\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:28.986092 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:28.986062    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf9p9\" (UniqueName: \"kubernetes.io/projected/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-kube-api-access-vf9p9\") pod \"e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") " pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:29.071729 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:29.071689    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:29.374869 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:29.374832    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"]
Oct 31 01:51:29.376887 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:51:29.376860    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed0a2f2_6812_4d37_b175_7ca1d14d5f9c.slice/crio-3d24aa07291b864fb41501c9d2ebd5e141c01d1bb07c490e7b923123c3dd477c WatchSource:0}: Error finding container 3d24aa07291b864fb41501c9d2ebd5e141c01d1bb07c490e7b923123c3dd477c: Status 404 returned error can't find the container with id 3d24aa07291b864fb41501c9d2ebd5e141c01d1bb07c490e7b923123c3dd477c
Oct 31 01:51:29.381513 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:29.381424    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:51:29.381732 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:29.381707    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:51:29.381890 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:29.381863    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:51:29.838865 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:29.838812    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq" event={"ID":"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c","Type":"ContainerStarted","Data":"af785e4b02c3d66b8906969382a903d60fe512552ec3a80ef4bf3e2667ec82e8"}
Oct 31 01:51:29.839075 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:29.838878    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq" event={"ID":"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c","Type":"ContainerStarted","Data":"3d24aa07291b864fb41501c9d2ebd5e141c01d1bb07c490e7b923123c3dd477c"}
Oct 31 01:51:29.864536 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:29.864489    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq" podStartSLOduration=1.864471946 podStartE2EDuration="1.864471946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:51:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:51:29.863225953 +0000 UTC m=+1845.866940247" watchObservedRunningTime="2026-10-31 01:51:29.864471946 +0000 UTC m=+1845.868186238"
Oct 31 01:51:30.072542 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:30.072506    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:30.079190 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:30.079161    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:30.848970 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:30.848938    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:30.849923 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:30.849897    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:51:32.438198 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.438168    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:51:32.596506 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.574009    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-podinfo\") pod \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") "
Oct 31 01:51:32.596506 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.574134    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-envoy\") pod \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") "
Oct 31 01:51:32.596506 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.574205    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istiod-ca-cert\") pod \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") "
Oct 31 01:51:32.596506 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.574242    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-data\") pod \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") "
Oct 31 01:51:32.596506 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.574278    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-token\" (UniqueName: \"kubernetes.io/projected/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-token\") pod \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") "
Oct 31 01:51:32.596506 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.574316    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-workload-socket\") pod \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") "
Oct 31 01:51:32.596506 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.574397    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-workload-certs\") pod \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") "
Oct 31 01:51:32.596506 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.574432    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-kube-api-access-s54xb\" (UniqueName: \"kubernetes.io/projected/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-kube-api-access-s54xb\") pod \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") "
Oct 31 01:51:32.596506 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.574511    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-credential-socket\") pod \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\" (UID: \"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99\") "
Oct 31 01:51:32.616517 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.597384    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-credential-socket" pod "e1f59db2-d04e-4e96-bf12-5a7d3f18eb99" (UID: "e1f59db2-d04e-4e96-bf12-5a7d3f18eb99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:32.616842 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.616812    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:32.617210 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.617186    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-data" pod "e1f59db2-d04e-4e96-bf12-5a7d3f18eb99" (UID: "e1f59db2-d04e-4e96-bf12-5a7d3f18eb99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:32.706526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.681223    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istiod-ca-cert" pod "e1f59db2-d04e-4e96-bf12-5a7d3f18eb99" (UID: "e1f59db2-d04e-4e96-bf12-5a7d3f18eb99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:32.706526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.681545    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-workload-certs" pod "e1f59db2-d04e-4e96-bf12-5a7d3f18eb99" (UID: "e1f59db2-d04e-4e96-bf12-5a7d3f18eb99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:32.706526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.696563    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-workload-socket" pod "e1f59db2-d04e-4e96-bf12-5a7d3f18eb99" (UID: "e1f59db2-d04e-4e96-bf12-5a7d3f18eb99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:32.741470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.740166    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:32.741470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.740223    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:32.741470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.740240    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:32.741470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.740255    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:32.783775 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.783735    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-token" pod "e1f59db2-d04e-4e96-bf12-5a7d3f18eb99" (UID: "e1f59db2-d04e-4e96-bf12-5a7d3f18eb99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:32.783921 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.783906    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-kube-api-access-s54xb" pod "e1f59db2-d04e-4e96-bf12-5a7d3f18eb99" (UID: "e1f59db2-d04e-4e96-bf12-5a7d3f18eb99"). InnerVolumeSpecName "kube-api-access-s54xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:32.785687 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.785655    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-envoy" pod "e1f59db2-d04e-4e96-bf12-5a7d3f18eb99" (UID: "e1f59db2-d04e-4e96-bf12-5a7d3f18eb99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:32.786293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.786262    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-podinfo" pod "e1f59db2-d04e-4e96-bf12-5a7d3f18eb99" (UID: "e1f59db2-d04e-4e96-bf12-5a7d3f18eb99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:51:32.856873 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.856831    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:32.857522 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.857493    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:32.857698 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.857667    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:32.857800 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.857786    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s54xb\" (UniqueName: \"kubernetes.io/projected/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99-kube-api-access-s54xb\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:32.870854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.870580    5650 generic.go:356] "Generic (PLEG): container finished" podID="e1f59db2-d04e-4e96-bf12-5a7d3f18eb99" containerID="5fae99081b3a102413ea91fc4d1a179fd86b3e1e3516cbe35c37748cebad01e0" exitCode=0
Oct 31 01:51:32.870854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.870657    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"
Oct 31 01:51:32.870854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.870774    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td" event={"ID":"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99","Type":"ContainerDied","Data":"5fae99081b3a102413ea91fc4d1a179fd86b3e1e3516cbe35c37748cebad01e0"}
Oct 31 01:51:32.870854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.870862    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td" event={"ID":"e1f59db2-d04e-4e96-bf12-5a7d3f18eb99","Type":"ContainerDied","Data":"7edf7de22c15edb070e4961486d1fc7e4f1082274b518b5de6ea351eca861e1b"}
Oct 31 01:51:32.871279 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.870887    5650 scope.go:122] "RemoveContainer" containerID="5fae99081b3a102413ea91fc4d1a179fd86b3e1e3516cbe35c37748cebad01e0"
Oct 31 01:51:32.883772 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.883748    5650 scope.go:122] "RemoveContainer" containerID="2f447df343bf3d68976ec259c95615222060b10afd976e1cefaabe5d3ddcf293"
Oct 31 01:51:32.894615 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.894565    5650 scope.go:122] "RemoveContainer" containerID="5fae99081b3a102413ea91fc4d1a179fd86b3e1e3516cbe35c37748cebad01e0"
Oct 31 01:51:32.895258 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:51:32.895148    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fae99081b3a102413ea91fc4d1a179fd86b3e1e3516cbe35c37748cebad01e0\": container with ID starting with 5fae99081b3a102413ea91fc4d1a179fd86b3e1e3516cbe35c37748cebad01e0 not found: ID does not exist" containerID="5fae99081b3a102413ea91fc4d1a179fd86b3e1e3516cbe35c37748cebad01e0"
Oct 31 01:51:32.895258 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.895187    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fae99081b3a102413ea91fc4d1a179fd86b3e1e3516cbe35c37748cebad01e0"} err="failed to get container status \"5fae99081b3a102413ea91fc4d1a179fd86b3e1e3516cbe35c37748cebad01e0\": rpc error: code = NotFound desc = could not find container \"5fae99081b3a102413ea91fc4d1a179fd86b3e1e3516cbe35c37748cebad01e0\": container with ID starting with 5fae99081b3a102413ea91fc4d1a179fd86b3e1e3516cbe35c37748cebad01e0 not found: ID does not exist"
Oct 31 01:51:32.895258 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.895211    5650 scope.go:122] "RemoveContainer" containerID="2f447df343bf3d68976ec259c95615222060b10afd976e1cefaabe5d3ddcf293"
Oct 31 01:51:32.895580 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:51:32.895556    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f447df343bf3d68976ec259c95615222060b10afd976e1cefaabe5d3ddcf293\": container with ID starting with 2f447df343bf3d68976ec259c95615222060b10afd976e1cefaabe5d3ddcf293 not found: ID does not exist" containerID="2f447df343bf3d68976ec259c95615222060b10afd976e1cefaabe5d3ddcf293"
Oct 31 01:51:32.895641 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.895589    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f447df343bf3d68976ec259c95615222060b10afd976e1cefaabe5d3ddcf293"} err="failed to get container status \"2f447df343bf3d68976ec259c95615222060b10afd976e1cefaabe5d3ddcf293\": rpc error: code = NotFound desc = could not find container \"2f447df343bf3d68976ec259c95615222060b10afd976e1cefaabe5d3ddcf293\": container with ID starting with 2f447df343bf3d68976ec259c95615222060b10afd976e1cefaabe5d3ddcf293 not found: ID does not exist"
Oct 31 01:51:32.899223 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.899198    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"]
Oct 31 01:51:32.944468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:32.943979    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e52ba1-openshift-default-845b6c8d4c-v78td"]
Oct 31 01:51:34.156494 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:34.156435    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1f59db2-d04e-4e96-bf12-5a7d3f18eb99" path="/var/lib/kubelet/pods/e1f59db2-d04e-4e96-bf12-5a7d3f18eb99/volumes"
Oct 31 01:51:34.888373 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:34.888341    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"]
Oct 31 01:51:34.888814 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:34.888784    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn" podUID="dcbe8798-c240-4556-bc46-d28d7ffacef2" containerName="istio-proxy" containerID="cri-o://09ae3a15b3a6aee49a89246a03cb493cbd3f522219b5bbdd1e087c64c3685cd0" gracePeriod=30
Oct 31 01:51:40.286793 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.286759    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:51:40.307739 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.307698    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-envoy\") pod \"dcbe8798-c240-4556-bc46-d28d7ffacef2\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") "
Oct 31 01:51:40.307910 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.307767    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-podinfo\") pod \"dcbe8798-c240-4556-bc46-d28d7ffacef2\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") "
Oct 31 01:51:40.307910 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.307898    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-token\" (UniqueName: \"kubernetes.io/projected/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-token\") pod \"dcbe8798-c240-4556-bc46-d28d7ffacef2\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") "
Oct 31 01:51:40.308018 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.307979    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcbe8798-c240-4556-bc46-d28d7ffacef2-kube-api-access-kf2ck\" (UniqueName: \"kubernetes.io/projected/dcbe8798-c240-4556-bc46-d28d7ffacef2-kube-api-access-kf2ck\") pod \"dcbe8798-c240-4556-bc46-d28d7ffacef2\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") "
Oct 31 01:51:40.308076 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.308029    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-data\") pod \"dcbe8798-c240-4556-bc46-d28d7ffacef2\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") "
Oct 31 01:51:40.308076 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.308062    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-credential-socket\") pod \"dcbe8798-c240-4556-bc46-d28d7ffacef2\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") "
Oct 31 01:51:40.308202 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.308183    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-workload-certs\") pod \"dcbe8798-c240-4556-bc46-d28d7ffacef2\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") "
Oct 31 01:51:40.308259 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.308232    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcbe8798-c240-4556-bc46-d28d7ffacef2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbe8798-c240-4556-bc46-d28d7ffacef2-istiod-ca-cert\") pod \"dcbe8798-c240-4556-bc46-d28d7ffacef2\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") "
Oct 31 01:51:40.308320 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.308263    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-workload-socket\") pod \"dcbe8798-c240-4556-bc46-d28d7ffacef2\" (UID: \"dcbe8798-c240-4556-bc46-d28d7ffacef2\") "
Oct 31 01:51:40.309319 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.309192    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-workload-certs" pod "dcbe8798-c240-4556-bc46-d28d7ffacef2" (UID: "dcbe8798-c240-4556-bc46-d28d7ffacef2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:40.309441 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.309360    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-workload-socket" pod "dcbe8798-c240-4556-bc46-d28d7ffacef2" (UID: "dcbe8798-c240-4556-bc46-d28d7ffacef2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:40.309606 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.309577    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-data" pod "dcbe8798-c240-4556-bc46-d28d7ffacef2" (UID: "dcbe8798-c240-4556-bc46-d28d7ffacef2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:40.309721 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.309639    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-credential-socket" pod "dcbe8798-c240-4556-bc46-d28d7ffacef2" (UID: "dcbe8798-c240-4556-bc46-d28d7ffacef2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:40.309797 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.309771    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbe8798-c240-4556-bc46-d28d7ffacef2-istiod-ca-cert" pod "dcbe8798-c240-4556-bc46-d28d7ffacef2" (UID: "dcbe8798-c240-4556-bc46-d28d7ffacef2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:40.312016 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.311520    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-envoy" pod "dcbe8798-c240-4556-bc46-d28d7ffacef2" (UID: "dcbe8798-c240-4556-bc46-d28d7ffacef2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:40.325575 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.314353    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbe8798-c240-4556-bc46-d28d7ffacef2-kube-api-access-kf2ck" pod "dcbe8798-c240-4556-bc46-d28d7ffacef2" (UID: "dcbe8798-c240-4556-bc46-d28d7ffacef2"). InnerVolumeSpecName "kube-api-access-kf2ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:40.327049 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.327006    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-token" pod "dcbe8798-c240-4556-bc46-d28d7ffacef2" (UID: "dcbe8798-c240-4556-bc46-d28d7ffacef2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:40.330363 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.329569    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-podinfo" pod "dcbe8798-c240-4556-bc46-d28d7ffacef2" (UID: "dcbe8798-c240-4556-bc46-d28d7ffacef2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:51:40.410379 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.410336    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:40.410379 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.410376    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbe8798-c240-4556-bc46-d28d7ffacef2-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:40.410379 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.410386    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:40.410691 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.410395    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:40.410691 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.410409    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:40.410691 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.410421    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:40.410691 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.410435    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf2ck\" (UniqueName: \"kubernetes.io/projected/dcbe8798-c240-4556-bc46-d28d7ffacef2-kube-api-access-kf2ck\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:40.410691 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.410462    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:40.410691 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.410475    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbe8798-c240-4556-bc46-d28d7ffacef2-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:51:40.601435 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.601345    5650 generic.go:356] "Generic (PLEG): container finished" podID="dcbe8798-c240-4556-bc46-d28d7ffacef2" containerID="09ae3a15b3a6aee49a89246a03cb493cbd3f522219b5bbdd1e087c64c3685cd0" exitCode=0
Oct 31 01:51:40.601647 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.601401    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn" event={"ID":"dcbe8798-c240-4556-bc46-d28d7ffacef2","Type":"ContainerDied","Data":"09ae3a15b3a6aee49a89246a03cb493cbd3f522219b5bbdd1e087c64c3685cd0"}
Oct 31 01:51:40.601694 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.601664    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn" event={"ID":"dcbe8798-c240-4556-bc46-d28d7ffacef2","Type":"ContainerDied","Data":"511d695da407518028b6ff617906a779518e1a4f045d98a7585ed2eb1d707c29"}
Oct 31 01:51:40.601741 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.601707    5650 scope.go:122] "RemoveContainer" containerID="09ae3a15b3a6aee49a89246a03cb493cbd3f522219b5bbdd1e087c64c3685cd0"
Oct 31 01:51:40.601836 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.601423    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"
Oct 31 01:51:40.622267 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.622244    5650 scope.go:122] "RemoveContainer" containerID="87d9887636d1211e30cfd8159074b990b8f029a0c3dc68a70ea34dd8ac4e0c01"
Oct 31 01:51:40.642051 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.642018    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"]
Oct 31 01:51:40.644049 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.644022    5650 scope.go:122] "RemoveContainer" containerID="09ae3a15b3a6aee49a89246a03cb493cbd3f522219b5bbdd1e087c64c3685cd0"
Oct 31 01:51:40.644430 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:51:40.644410    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09ae3a15b3a6aee49a89246a03cb493cbd3f522219b5bbdd1e087c64c3685cd0\": container with ID starting with 09ae3a15b3a6aee49a89246a03cb493cbd3f522219b5bbdd1e087c64c3685cd0 not found: ID does not exist" containerID="09ae3a15b3a6aee49a89246a03cb493cbd3f522219b5bbdd1e087c64c3685cd0"
Oct 31 01:51:40.644531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.644439    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09ae3a15b3a6aee49a89246a03cb493cbd3f522219b5bbdd1e087c64c3685cd0"} err="failed to get container status \"09ae3a15b3a6aee49a89246a03cb493cbd3f522219b5bbdd1e087c64c3685cd0\": rpc error: code = NotFound desc = could not find container \"09ae3a15b3a6aee49a89246a03cb493cbd3f522219b5bbdd1e087c64c3685cd0\": container with ID starting with 09ae3a15b3a6aee49a89246a03cb493cbd3f522219b5bbdd1e087c64c3685cd0 not found: ID does not exist"
Oct 31 01:51:40.644531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.644506    5650 scope.go:122] "RemoveContainer" containerID="87d9887636d1211e30cfd8159074b990b8f029a0c3dc68a70ea34dd8ac4e0c01"
Oct 31 01:51:40.644866 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:51:40.644842    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d9887636d1211e30cfd8159074b990b8f029a0c3dc68a70ea34dd8ac4e0c01\": container with ID starting with 87d9887636d1211e30cfd8159074b990b8f029a0c3dc68a70ea34dd8ac4e0c01 not found: ID does not exist" containerID="87d9887636d1211e30cfd8159074b990b8f029a0c3dc68a70ea34dd8ac4e0c01"
Oct 31 01:51:40.644950 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.644883    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d9887636d1211e30cfd8159074b990b8f029a0c3dc68a70ea34dd8ac4e0c01"} err="failed to get container status \"87d9887636d1211e30cfd8159074b990b8f029a0c3dc68a70ea34dd8ac4e0c01\": rpc error: code = NotFound desc = could not find container \"87d9887636d1211e30cfd8159074b990b8f029a0c3dc68a70ea34dd8ac4e0c01\": container with ID starting with 87d9887636d1211e30cfd8159074b990b8f029a0c3dc68a70ea34dd8ac4e0c01 not found: ID does not exist"
Oct 31 01:51:40.659968 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.659929    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-90c4d1-openshift-default-ddcf89cb7-zj6cn"]
Oct 31 01:51:40.941387 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.941301    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt"]
Oct 31 01:51:40.942240 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.942214    5650 cpu_manager.go:414] "containerMap: removing container" podUID="e1f59db2-d04e-4e96-bf12-5a7d3f18eb99" containerName="istio-proxy"
Oct 31 01:51:40.942240 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.942239    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1f59db2-d04e-4e96-bf12-5a7d3f18eb99" containerName="istio-proxy"
Oct 31 01:51:40.942432 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.942273    5650 cpu_manager.go:414] "containerMap: removing container" podUID="dcbe8798-c240-4556-bc46-d28d7ffacef2" containerName="istio-proxy"
Oct 31 01:51:40.942432 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.942318    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcbe8798-c240-4556-bc46-d28d7ffacef2" containerName="istio-proxy"
Oct 31 01:51:40.942432 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.942376    5650 cpu_manager.go:414] "containerMap: removing container" podUID="dcbe8798-c240-4556-bc46-d28d7ffacef2" containerName="istio-proxy"
Oct 31 01:51:40.942432 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.942385    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcbe8798-c240-4556-bc46-d28d7ffacef2" containerName="istio-proxy"
Oct 31 01:51:40.942689 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.942559    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1f59db2-d04e-4e96-bf12-5a7d3f18eb99" containerName="istio-proxy"
Oct 31 01:51:40.942689 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.942573    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcbe8798-c240-4556-bc46-d28d7ffacef2" containerName="istio-proxy"
Oct 31 01:51:40.942689 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:40.942586    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcbe8798-c240-4556-bc46-d28d7ffacef2" containerName="istio-proxy"
Oct 31 01:51:41.036730 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:41.036683    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt"]
Oct 31 01:51:41.036913 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:41.036828    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt"
Oct 31 01:51:41.044220 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:41.042770    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a3ff0c-serving-cert\""
Oct 31 01:51:41.122631 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:41.122593    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/548329c1-14a7-4efe-844f-248210658f41-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt\" (UID: \"548329c1-14a7-4efe-844f-248210658f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt"
Oct 31 01:51:41.122897 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:41.122728    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/548329c1-14a7-4efe-844f-248210658f41-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt\" (UID: \"548329c1-14a7-4efe-844f-248210658f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt"
Oct 31 01:51:41.122897 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:41.122768    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft6d4\" (UniqueName: \"kubernetes.io/projected/548329c1-14a7-4efe-844f-248210658f41-kube-api-access-ft6d4\") pod \"maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt\" (UID: \"548329c1-14a7-4efe-844f-248210658f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt"
Oct 31 01:51:41.123023 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:41.122966    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/548329c1-14a7-4efe-844f-248210658f41-metrics-certs\") pod \"maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt\" (UID: \"548329c1-14a7-4efe-844f-248210658f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt"
Oct 31 01:51:41.224964 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:41.224926    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/548329c1-14a7-4efe-844f-248210658f41-metrics-certs\") pod \"maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt\" (UID: \"548329c1-14a7-4efe-844f-248210658f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt"
Oct 31 01:51:41.225179 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:41.225037    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/548329c1-14a7-4efe-844f-248210658f41-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt\" (UID: \"548329c1-14a7-4efe-844f-248210658f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt"
Oct 31 01:51:41.225251 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:41.225218    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/548329c1-14a7-4efe-844f-248210658f41-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt\" (UID: \"548329c1-14a7-4efe-844f-248210658f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt"
Oct 31 01:51:41.225315 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:41.225282    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft6d4\" (UniqueName: \"kubernetes.io/projected/548329c1-14a7-4efe-844f-248210658f41-kube-api-access-ft6d4\") pod \"maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt\" (UID: \"548329c1-14a7-4efe-844f-248210658f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt"
Oct 31 01:51:41.225864 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:41.225835    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/548329c1-14a7-4efe-844f-248210658f41-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt\" (UID: \"548329c1-14a7-4efe-844f-248210658f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt"
Oct 31 01:51:41.227728 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:41.227702    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/548329c1-14a7-4efe-844f-248210658f41-metrics-certs\") pod \"maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt\" (UID: \"548329c1-14a7-4efe-844f-248210658f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt"
Oct 31 01:51:41.228181 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:41.228157    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/548329c1-14a7-4efe-844f-248210658f41-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt\" (UID: \"548329c1-14a7-4efe-844f-248210658f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt"
Oct 31 01:51:41.242280 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:41.242249    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft6d4\" (UniqueName: \"kubernetes.io/projected/548329c1-14a7-4efe-844f-248210658f41-kube-api-access-ft6d4\") pod \"maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt\" (UID: \"548329c1-14a7-4efe-844f-248210658f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt"
Oct 31 01:51:41.392626 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:41.392581    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt"
Oct 31 01:51:41.788533 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:41.788491    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt"]
Oct 31 01:51:41.792891 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:51:41.792856    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548329c1_14a7_4efe_844f_248210658f41.slice/crio-90e4cdfcaf0940a2fd670d85adc9ab541a8a739caa5d3b3f26a4cadedfcfb36a WatchSource:0}: Error finding container 90e4cdfcaf0940a2fd670d85adc9ab541a8a739caa5d3b3f26a4cadedfcfb36a: Status 404 returned error can't find the container with id 90e4cdfcaf0940a2fd670d85adc9ab541a8a739caa5d3b3f26a4cadedfcfb36a
Oct 31 01:51:42.126096 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.126052    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcbe8798-c240-4556-bc46-d28d7ffacef2" path="/var/lib/kubelet/pods/dcbe8798-c240-4556-bc46-d28d7ffacef2/volumes"
Oct 31 01:51:42.662916 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.653046    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt" event={"ID":"548329c1-14a7-4efe-844f-248210658f41","Type":"ContainerStarted","Data":"7c40721e028a9baa4a23bdb0e0e3bf8dc7ad203e77c7faa016b4555fbddd25a4"}
Oct 31 01:51:42.662916 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.653095    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt" event={"ID":"548329c1-14a7-4efe-844f-248210658f41","Type":"ContainerStarted","Data":"90e4cdfcaf0940a2fd670d85adc9ab541a8a739caa5d3b3f26a4cadedfcfb36a"}
Oct 31 01:51:42.662916 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.653295    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt"
Oct 31 01:51:42.941592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.677729    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt" podStartSLOduration=2.097208302 podStartE2EDuration="2.677714114s" totalImagesPullingTime="580.505812ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:51:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:51:42.676576447 +0000 UTC m=+1858.680290740" watchObservedRunningTime="2026-10-31 01:51:42.677714114 +0000 UTC m=+1858.681428405"
Oct 31 01:51:42.941592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.687237    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp"]
Oct 31 01:51:42.941592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.687984    5650 cpu_manager.go:414] "containerMap: removing container" podUID="e1f59db2-d04e-4e96-bf12-5a7d3f18eb99" containerName="istio-proxy"
Oct 31 01:51:42.941592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.687999    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1f59db2-d04e-4e96-bf12-5a7d3f18eb99" containerName="istio-proxy"
Oct 31 01:51:42.941592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.688202    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1f59db2-d04e-4e96-bf12-5a7d3f18eb99" containerName="istio-proxy"
Oct 31 01:51:42.941592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.705844    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t"]
Oct 31 01:51:42.941592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.709977    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp"
Oct 31 01:51:42.941592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.720182    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-b8d890-dockercfg-sllxr\""
Oct 31 01:51:42.941592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.720182    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-b8d890\""
Oct 31 01:51:42.941592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.734252    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp"]
Oct 31 01:51:42.941592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.734298    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t"]
Oct 31 01:51:42.941592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.734395    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t"
Oct 31 01:51:42.941592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.758295    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7cdd7c3-0ba1-4558-8043-1be465a26b46-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp\" (UID: \"f7cdd7c3-0ba1-4558-8043-1be465a26b46\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp"
Oct 31 01:51:42.941592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.758406    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jrsp\" (UniqueName: \"kubernetes.io/projected/f7cdd7c3-0ba1-4558-8043-1be465a26b46-kube-api-access-4jrsp\") pod \"payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp\" (UID: \"f7cdd7c3-0ba1-4558-8043-1be465a26b46\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp"
Oct 31 01:51:42.941592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.861131    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7ds6\" (UniqueName: \"kubernetes.io/projected/1a64682c-f620-4031-8efd-d576b36a9161-kube-api-access-b7ds6\") pod \"payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t\" (UID: \"1a64682c-f620-4031-8efd-d576b36a9161\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t"
Oct 31 01:51:42.941592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.861301    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7cdd7c3-0ba1-4558-8043-1be465a26b46-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp\" (UID: \"f7cdd7c3-0ba1-4558-8043-1be465a26b46\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp"
Oct 31 01:51:42.941592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.861343    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a64682c-f620-4031-8efd-d576b36a9161-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t\" (UID: \"1a64682c-f620-4031-8efd-d576b36a9161\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t"
Oct 31 01:51:42.941592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.861387    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jrsp\" (UniqueName: \"kubernetes.io/projected/f7cdd7c3-0ba1-4558-8043-1be465a26b46-kube-api-access-4jrsp\") pod \"payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp\" (UID: \"f7cdd7c3-0ba1-4558-8043-1be465a26b46\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp"
Oct 31 01:51:42.941592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.862276    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7cdd7c3-0ba1-4558-8043-1be465a26b46-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp\" (UID: \"f7cdd7c3-0ba1-4558-8043-1be465a26b46\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp"
Oct 31 01:51:42.941592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.876671    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jrsp\" (UniqueName: \"kubernetes.io/projected/f7cdd7c3-0ba1-4558-8043-1be465a26b46-kube-api-access-4jrsp\") pod \"payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp\" (UID: \"f7cdd7c3-0ba1-4558-8043-1be465a26b46\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp"
Oct 31 01:51:42.962878 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.962841    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7ds6\" (UniqueName: \"kubernetes.io/projected/1a64682c-f620-4031-8efd-d576b36a9161-kube-api-access-b7ds6\") pod \"payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t\" (UID: \"1a64682c-f620-4031-8efd-d576b36a9161\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t"
Oct 31 01:51:42.963080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.962956    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a64682c-f620-4031-8efd-d576b36a9161-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t\" (UID: \"1a64682c-f620-4031-8efd-d576b36a9161\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t"
Oct 31 01:51:42.963659 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.963635    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a64682c-f620-4031-8efd-d576b36a9161-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t\" (UID: \"1a64682c-f620-4031-8efd-d576b36a9161\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t"
Oct 31 01:51:42.976334 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:42.976303    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7ds6\" (UniqueName: \"kubernetes.io/projected/1a64682c-f620-4031-8efd-d576b36a9161-kube-api-access-b7ds6\") pod \"payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t\" (UID: \"1a64682c-f620-4031-8efd-d576b36a9161\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t"
Oct 31 01:51:43.054630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:43.054595    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp"
Oct 31 01:51:43.079744 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:43.079718    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t"
Oct 31 01:51:43.566175 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:51:43.553196    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7cdd7c3_0ba1_4558_8043_1be465a26b46.slice/crio-4130bdf0df30889ed723b65ee0338d95351cd24273382641e9d9a6bf1091c231 WatchSource:0}: Error finding container 4130bdf0df30889ed723b65ee0338d95351cd24273382641e9d9a6bf1091c231: Status 404 returned error can't find the container with id 4130bdf0df30889ed723b65ee0338d95351cd24273382641e9d9a6bf1091c231
Oct 31 01:51:43.566175 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:43.553637    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp"]
Oct 31 01:51:43.631290 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:51:43.631247    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a64682c_f620_4031_8efd_d576b36a9161.slice/crio-37810afd152d95b154087daa18673a73a3947173ef4227a4a5119f9e8f3babb1 WatchSource:0}: Error finding container 37810afd152d95b154087daa18673a73a3947173ef4227a4a5119f9e8f3babb1: Status 404 returned error can't find the container with id 37810afd152d95b154087daa18673a73a3947173ef4227a4a5119f9e8f3babb1
Oct 31 01:51:43.633320 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:43.633296    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t"]
Oct 31 01:51:43.686891 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:43.686861    5650 generic.go:356] "Generic (PLEG): container finished" podID="fec61f6a-7103-49ef-ae2c-bbe826dcd159" containerID="b279e1bd517d054265149f6052141ba7260ab6f000c754a18f21c3aa903f9f2c" exitCode=0
Oct 31 01:51:43.687324 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:43.686938    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t" event={"ID":"fec61f6a-7103-49ef-ae2c-bbe826dcd159","Type":"ContainerDied","Data":"b279e1bd517d054265149f6052141ba7260ab6f000c754a18f21c3aa903f9f2c"}
Oct 31 01:51:43.687502 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:43.687439    5650 scope.go:122] "RemoveContainer" containerID="b279e1bd517d054265149f6052141ba7260ab6f000c754a18f21c3aa903f9f2c"
Oct 31 01:51:43.701933 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:43.699629    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp" event={"ID":"f7cdd7c3-0ba1-4558-8043-1be465a26b46","Type":"ContainerStarted","Data":"48b04a17852251e51d0298f8a09e4a4ca39c981d925561fea5cc7b385fd6fe57"}
Oct 31 01:51:43.701933 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:43.699692    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp" event={"ID":"f7cdd7c3-0ba1-4558-8043-1be465a26b46","Type":"ContainerStarted","Data":"4130bdf0df30889ed723b65ee0338d95351cd24273382641e9d9a6bf1091c231"}
Oct 31 01:51:43.701933 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:43.700433    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp"
Oct 31 01:51:43.701933 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:43.701802    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:51:43.701933 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:43.701884    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:51:43.702342 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:43.701946    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:51:43.712350 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:43.712253    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t" event={"ID":"1a64682c-f620-4031-8efd-d576b36a9161","Type":"ContainerStarted","Data":"37810afd152d95b154087daa18673a73a3947173ef4227a4a5119f9e8f3babb1"}
Oct 31 01:51:43.764438 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:43.764373    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp" podStartSLOduration=1.764353964 podStartE2EDuration="1.764353964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:51:43.761020891 +0000 UTC m=+1859.764735194" watchObservedRunningTime="2026-10-31 01:51:43.764353964 +0000 UTC m=+1859.768068255"
Oct 31 01:51:44.734733 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:44.734696    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t" event={"ID":"1a64682c-f620-4031-8efd-d576b36a9161","Type":"ContainerStarted","Data":"3706c689facde2b43b828d78bafd6824b466cefeeffe2bc85339c565daac52cb"}
Oct 31 01:51:44.735243 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:44.734781    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t"
Oct 31 01:51:44.736947 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:44.736921    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t" event={"ID":"fec61f6a-7103-49ef-ae2c-bbe826dcd159","Type":"ContainerStarted","Data":"9e35bf6df1a1ed5a89453788b676949f6836d64ce36d0d0ee40c15ecffb44d20"}
Oct 31 01:51:44.738925 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:44.738900    5650 generic.go:356] "Generic (PLEG): container finished" podID="04d96b2a-ab79-4c0a-a187-766ef1a58c0f" containerID="7c31e35ad7c443b677c136f4d46b61312e5fabee20541ea266b9eb982ada8732" exitCode=0
Oct 31 01:51:44.739047 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:44.738978    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr" event={"ID":"04d96b2a-ab79-4c0a-a187-766ef1a58c0f","Type":"ContainerDied","Data":"7c31e35ad7c443b677c136f4d46b61312e5fabee20541ea266b9eb982ada8732"}
Oct 31 01:51:44.739606 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:44.739568    5650 scope.go:122] "RemoveContainer" containerID="7c31e35ad7c443b677c136f4d46b61312e5fabee20541ea266b9eb982ada8732"
Oct 31 01:51:44.741088 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:44.741062    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:51:44.741155 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:44.741136    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:51:44.741194 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:44.741169    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:51:44.758239 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:44.758189    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t" podStartSLOduration=2.7581717709999998 podStartE2EDuration="2.758171771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:51:44.756180549 +0000 UTC m=+1860.759894833" watchObservedRunningTime="2026-10-31 01:51:44.758171771 +0000 UTC m=+1860.761886063"
Oct 31 01:51:44.765244 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:44.765222    5650 scope.go:122] "RemoveContainer" containerID="9ef4f1cfb18482bfc29e1cdf32fb054ddffd64bbac8280dfcf8b3e5fc15840c2"
Oct 31 01:51:44.778694 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:44.778676    5650 scope.go:122] "RemoveContainer" containerID="7caafb5aa8ee71535cb85a9b8101d9056491496f8095cb29f3824bee2e190e8f"
Oct 31 01:51:45.554996 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:45.554959    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:51:45.556376 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:45.556349    5650 patch_prober.go:29] interesting pod/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Oct 31 01:51:45.556547 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:45.556392    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t" podUID="fec61f6a-7103-49ef-ae2c-bbe826dcd159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Oct 31 01:51:45.780757 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:45.780717    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr" event={"ID":"04d96b2a-ab79-4c0a-a187-766ef1a58c0f","Type":"ContainerStarted","Data":"bc4c12272e6738a3da9ca1a38aceabd35b8325415641631fda9499f20073ad94"}
Oct 31 01:51:46.331671 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:46.331631    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:51:46.548474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:46.546156    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:51:46.548474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:46.547073    5650 patch_prober.go:29] interesting pod/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 31 01:51:46.548474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:46.547128    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr" podUID="04d96b2a-ab79-4c0a-a187-766ef1a58c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 31 01:51:46.554904 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:46.554867    5650 patch_prober.go:29] interesting pod/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Oct 31 01:51:46.555102 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:46.554916    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t" podUID="fec61f6a-7103-49ef-ae2c-bbe826dcd159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Oct 31 01:51:47.354471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:47.348946    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:51:47.569303 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:47.569268    5650 patch_prober.go:29] interesting pod/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Oct 31 01:51:47.569639 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:47.569602    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t" podUID="fec61f6a-7103-49ef-ae2c-bbe826dcd159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Oct 31 01:51:47.570266 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:47.570243    5650 patch_prober.go:29] interesting pod/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 31 01:51:47.570420 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:47.570400    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr" podUID="04d96b2a-ab79-4c0a-a187-766ef1a58c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 31 01:51:48.554525 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:48.553756    5650 patch_prober.go:29] interesting pod/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 31 01:51:48.554525 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:48.553833    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr" podUID="04d96b2a-ab79-4c0a-a187-766ef1a58c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 31 01:51:48.555569 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:48.555538    5650 patch_prober.go:29] interesting pod/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Oct 31 01:51:48.555812 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:48.555786    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t" podUID="fec61f6a-7103-49ef-ae2c-bbe826dcd159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Oct 31 01:51:48.757686 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:48.757649    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt"
Oct 31 01:51:49.538676 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:49.538638    5650 patch_prober.go:29] interesting pod/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 31 01:51:49.538992 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:49.538961    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr" podUID="04d96b2a-ab79-4c0a-a187-766ef1a58c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 31 01:51:49.555294 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:49.555260    5650 patch_prober.go:29] interesting pod/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Oct 31 01:51:49.555788 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:49.555762    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t" podUID="fec61f6a-7103-49ef-ae2c-bbe826dcd159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Oct 31 01:51:50.535771 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:50.535731    5650 patch_prober.go:29] interesting pod/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 31 01:51:50.535954 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:50.535791    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr" podUID="04d96b2a-ab79-4c0a-a187-766ef1a58c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 31 01:51:50.559830 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:50.559797    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:51:50.561026 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:50.561001    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:51:51.542202 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:51.542168    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:51:51.543522 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:51.543500    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:51:54.742565 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:54.742529    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp"
Oct 31 01:51:55.784142 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:51:55.784111    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t"
Oct 31 01:51:58.621466 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:51:58.617951    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:52:03.944039 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:03.944006    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp"]
Oct 31 01:52:03.944755 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:03.944694    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp" podUID="b2342b77-070b-4cbb-9856-f0f075c05719" containerName="maas-api" containerID="cri-o://e7d9d94e9db481a3ce5a07ed4f503c62ea599efb1c9ac96351b258b370230cda" gracePeriod=30
Oct 31 01:52:03.988748 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:03.988706    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7"]
Oct 31 01:52:04.029117 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:04.028509    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs"]
Oct 31 01:52:04.646130 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:04.645749    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp"
Oct 31 01:52:04.794811 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:04.794776    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2342b77-070b-4cbb-9856-f0f075c05719-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2342b77-070b-4cbb-9856-f0f075c05719-openshift-service-ca\") pod \"b2342b77-070b-4cbb-9856-f0f075c05719\" (UID: \"b2342b77-070b-4cbb-9856-f0f075c05719\") "
Oct 31 01:52:04.794968 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:04.794929    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2342b77-070b-4cbb-9856-f0f075c05719-kube-api-access-c8gqt\" (UniqueName: \"kubernetes.io/projected/b2342b77-070b-4cbb-9856-f0f075c05719-kube-api-access-c8gqt\") pod \"b2342b77-070b-4cbb-9856-f0f075c05719\" (UID: \"b2342b77-070b-4cbb-9856-f0f075c05719\") "
Oct 31 01:52:04.795062 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:04.795047    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2342b77-070b-4cbb-9856-f0f075c05719-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2342b77-070b-4cbb-9856-f0f075c05719-maas-api-tls\") pod \"b2342b77-070b-4cbb-9856-f0f075c05719\" (UID: \"b2342b77-070b-4cbb-9856-f0f075c05719\") "
Oct 31 01:52:04.795103 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:04.795083    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2342b77-070b-4cbb-9856-f0f075c05719-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2342b77-070b-4cbb-9856-f0f075c05719-metrics-certs\") pod \"b2342b77-070b-4cbb-9856-f0f075c05719\" (UID: \"b2342b77-070b-4cbb-9856-f0f075c05719\") "
Oct 31 01:52:04.795153 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:04.795131    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2342b77-070b-4cbb-9856-f0f075c05719-openshift-service-ca" pod "b2342b77-070b-4cbb-9856-f0f075c05719" (UID: "b2342b77-070b-4cbb-9856-f0f075c05719"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:04.795778 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:04.795756    5650 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2342b77-070b-4cbb-9856-f0f075c05719-openshift-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:04.797222 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:04.797193    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2342b77-070b-4cbb-9856-f0f075c05719-metrics-certs" pod "b2342b77-070b-4cbb-9856-f0f075c05719" (UID: "b2342b77-070b-4cbb-9856-f0f075c05719"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:52:04.797325 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:04.797224    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2342b77-070b-4cbb-9856-f0f075c05719-maas-api-tls" pod "b2342b77-070b-4cbb-9856-f0f075c05719" (UID: "b2342b77-070b-4cbb-9856-f0f075c05719"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:52:04.797325 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:04.797287    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2342b77-070b-4cbb-9856-f0f075c05719-kube-api-access-c8gqt" pod "b2342b77-070b-4cbb-9856-f0f075c05719" (UID: "b2342b77-070b-4cbb-9856-f0f075c05719"). InnerVolumeSpecName "kube-api-access-c8gqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:04.897682 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:04.897643    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8gqt\" (UniqueName: \"kubernetes.io/projected/b2342b77-070b-4cbb-9856-f0f075c05719-kube-api-access-c8gqt\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:04.897849 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:04.897698    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2342b77-070b-4cbb-9856-f0f075c05719-maas-api-tls\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:04.897849 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:04.897708    5650 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2342b77-070b-4cbb-9856-f0f075c05719-metrics-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:05.041730 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:05.041693    5650 generic.go:356] "Generic (PLEG): container finished" podID="b2342b77-070b-4cbb-9856-f0f075c05719" containerID="e7d9d94e9db481a3ce5a07ed4f503c62ea599efb1c9ac96351b258b370230cda" exitCode=0
Oct 31 01:52:05.042190 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:05.041744    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp"
Oct 31 01:52:05.042190 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:05.041758    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp" event={"ID":"b2342b77-070b-4cbb-9856-f0f075c05719","Type":"ContainerDied","Data":"e7d9d94e9db481a3ce5a07ed4f503c62ea599efb1c9ac96351b258b370230cda"}
Oct 31 01:52:05.042190 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:05.041800    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp" event={"ID":"b2342b77-070b-4cbb-9856-f0f075c05719","Type":"ContainerDied","Data":"1adcbb581ef6ddb12c9c6ac2d78bbd7775e4d8d639bf52afb1c93920d3261116"}
Oct 31 01:52:05.042190 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:05.041821    5650 scope.go:122] "RemoveContainer" containerID="e7d9d94e9db481a3ce5a07ed4f503c62ea599efb1c9ac96351b258b370230cda"
Oct 31 01:52:05.054510 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:05.054429    5650 scope.go:122] "RemoveContainer" containerID="e7d9d94e9db481a3ce5a07ed4f503c62ea599efb1c9ac96351b258b370230cda"
Oct 31 01:52:05.055077 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:52:05.055048    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7d9d94e9db481a3ce5a07ed4f503c62ea599efb1c9ac96351b258b370230cda\": container with ID starting with e7d9d94e9db481a3ce5a07ed4f503c62ea599efb1c9ac96351b258b370230cda not found: ID does not exist" containerID="e7d9d94e9db481a3ce5a07ed4f503c62ea599efb1c9ac96351b258b370230cda"
Oct 31 01:52:05.055179 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:05.055087    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7d9d94e9db481a3ce5a07ed4f503c62ea599efb1c9ac96351b258b370230cda"} err="failed to get container status \"e7d9d94e9db481a3ce5a07ed4f503c62ea599efb1c9ac96351b258b370230cda\": rpc error: code = NotFound desc = could not find container \"e7d9d94e9db481a3ce5a07ed4f503c62ea599efb1c9ac96351b258b370230cda\": container with ID starting with e7d9d94e9db481a3ce5a07ed4f503c62ea599efb1c9ac96351b258b370230cda not found: ID does not exist"
Oct 31 01:52:05.076108 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:05.076023    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp"]
Oct 31 01:52:05.081799 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:05.081763    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c06ae4-7c8cbd45f8-879sp"]
Oct 31 01:52:06.120948 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:06.120908    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2342b77-070b-4cbb-9856-f0f075c05719" path="/var/lib/kubelet/pods/b2342b77-070b-4cbb-9856-f0f075c05719/volumes"
Oct 31 01:52:07.719540 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:07.719505    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft"]
Oct 31 01:52:07.720215 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:07.720197    5650 cpu_manager.go:414] "containerMap: removing container" podUID="b2342b77-070b-4cbb-9856-f0f075c05719" containerName="maas-api"
Oct 31 01:52:07.720215 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:07.720214    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2342b77-070b-4cbb-9856-f0f075c05719" containerName="maas-api"
Oct 31 01:52:07.720359 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:07.720322    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2342b77-070b-4cbb-9856-f0f075c05719" containerName="maas-api"
Oct 31 01:52:07.736328 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:07.736296    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft"
Oct 31 01:52:07.738125 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:07.738095    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft"]
Oct 31 01:52:07.742506 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:07.742268    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf"]
Oct 31 01:52:07.745928 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:07.744333    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-278a56\""
Oct 31 01:52:07.745928 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:07.744401    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-278a56-dockercfg-tcf9s\""
Oct 31 01:52:07.757737 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:07.757704    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf"
Oct 31 01:52:07.769746 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:07.769710    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf"]
Oct 31 01:52:07.854766 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:07.854720    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/786fd80e-b891-4ccc-a776-069b6f6ba3f9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf\" (UID: \"786fd80e-b891-4ccc-a776-069b6f6ba3f9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf"
Oct 31 01:52:07.854995 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:07.854869    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltj7s\" (UniqueName: \"kubernetes.io/projected/22a469a4-b8b4-4601-a954-a992c76be571-kube-api-access-ltj7s\") pod \"payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft\" (UID: \"22a469a4-b8b4-4601-a954-a992c76be571\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft"
Oct 31 01:52:07.854995 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:07.854982    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d559k\" (UniqueName: \"kubernetes.io/projected/786fd80e-b891-4ccc-a776-069b6f6ba3f9-kube-api-access-d559k\") pod \"payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf\" (UID: \"786fd80e-b891-4ccc-a776-069b6f6ba3f9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf"
Oct 31 01:52:07.855121 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:07.855056    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22a469a4-b8b4-4601-a954-a992c76be571-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft\" (UID: \"22a469a4-b8b4-4601-a954-a992c76be571\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft"
Oct 31 01:52:07.956332 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:07.956290    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/786fd80e-b891-4ccc-a776-069b6f6ba3f9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf\" (UID: \"786fd80e-b891-4ccc-a776-069b6f6ba3f9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf"
Oct 31 01:52:07.956555 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:07.956406    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltj7s\" (UniqueName: \"kubernetes.io/projected/22a469a4-b8b4-4601-a954-a992c76be571-kube-api-access-ltj7s\") pod \"payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft\" (UID: \"22a469a4-b8b4-4601-a954-a992c76be571\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft"
Oct 31 01:52:07.956555 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:07.956541    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d559k\" (UniqueName: \"kubernetes.io/projected/786fd80e-b891-4ccc-a776-069b6f6ba3f9-kube-api-access-d559k\") pod \"payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf\" (UID: \"786fd80e-b891-4ccc-a776-069b6f6ba3f9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf"
Oct 31 01:52:07.956697 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:07.956614    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22a469a4-b8b4-4601-a954-a992c76be571-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft\" (UID: \"22a469a4-b8b4-4601-a954-a992c76be571\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft"
Oct 31 01:52:07.957167 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:07.957145    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/786fd80e-b891-4ccc-a776-069b6f6ba3f9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf\" (UID: \"786fd80e-b891-4ccc-a776-069b6f6ba3f9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf"
Oct 31 01:52:07.957380 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:07.957362    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22a469a4-b8b4-4601-a954-a992c76be571-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft\" (UID: \"22a469a4-b8b4-4601-a954-a992c76be571\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft"
Oct 31 01:52:07.970305 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:07.970225    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d559k\" (UniqueName: \"kubernetes.io/projected/786fd80e-b891-4ccc-a776-069b6f6ba3f9-kube-api-access-d559k\") pod \"payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf\" (UID: \"786fd80e-b891-4ccc-a776-069b6f6ba3f9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf"
Oct 31 01:52:07.970478 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:07.970345    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltj7s\" (UniqueName: \"kubernetes.io/projected/22a469a4-b8b4-4601-a954-a992c76be571-kube-api-access-ltj7s\") pod \"payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft\" (UID: \"22a469a4-b8b4-4601-a954-a992c76be571\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft"
Oct 31 01:52:08.080664 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:08.080630    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft"
Oct 31 01:52:08.101767 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:08.101735    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf"
Oct 31 01:52:08.405436 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:52:08.405407    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a469a4_b8b4_4601_a954_a992c76be571.slice/crio-45985ffc0e82522aa529019c34ac3cfaed77b62cbab66af70f6baf1179f7dceb WatchSource:0}: Error finding container 45985ffc0e82522aa529019c34ac3cfaed77b62cbab66af70f6baf1179f7dceb: Status 404 returned error can't find the container with id 45985ffc0e82522aa529019c34ac3cfaed77b62cbab66af70f6baf1179f7dceb
Oct 31 01:52:08.405586 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:08.405520    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft"]
Oct 31 01:52:08.453968 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:08.453939    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf"]
Oct 31 01:52:08.459879 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:52:08.459846    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod786fd80e_b891_4ccc_a776_069b6f6ba3f9.slice/crio-0663cfcff8de6e27b734bed7d2e77dee1004e2e362f615cc1dac84b4d2aea821 WatchSource:0}: Error finding container 0663cfcff8de6e27b734bed7d2e77dee1004e2e362f615cc1dac84b4d2aea821: Status 404 returned error can't find the container with id 0663cfcff8de6e27b734bed7d2e77dee1004e2e362f615cc1dac84b4d2aea821
Oct 31 01:52:08.989765 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:08.989723    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7" podUID="c0ab4857-a9ba-4524-9025-d2906539f9c3" containerName="payload-processing" containerID="cri-o://7219246d9bb388351b0d5d7cdb622128a1c1b680cf3ac00f0551b6bace33a7ad" gracePeriod=25
Oct 31 01:52:09.029544 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:09.029474    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs" podUID="14a414fd-b21c-4086-bae8-090b9c22ba38" containerName="payload-pre-processing" containerID="cri-o://747f8d17c5dc8f293b5277bebae773ffc92deb039338fbf9bc5ae6fb1d588962" gracePeriod=25
Oct 31 01:52:09.082943 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:09.082902    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf" event={"ID":"786fd80e-b891-4ccc-a776-069b6f6ba3f9","Type":"ContainerStarted","Data":"aad3384e27760598dc81cc9262ec53bf8236bcd4a61d40f3cc16910cb7d5eabe"}
Oct 31 01:52:09.083140 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:09.082964    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf" event={"ID":"786fd80e-b891-4ccc-a776-069b6f6ba3f9","Type":"ContainerStarted","Data":"0663cfcff8de6e27b734bed7d2e77dee1004e2e362f615cc1dac84b4d2aea821"}
Oct 31 01:52:09.083140 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:09.083029    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf"
Oct 31 01:52:09.084961 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:09.084929    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft" event={"ID":"22a469a4-b8b4-4601-a954-a992c76be571","Type":"ContainerStarted","Data":"5a21a20c2f28ed5c46d99e3b0116a73d83902da91ae618de611cf55bbab1f83a"}
Oct 31 01:52:09.085076 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:09.084974    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft" event={"ID":"22a469a4-b8b4-4601-a954-a992c76be571","Type":"ContainerStarted","Data":"45985ffc0e82522aa529019c34ac3cfaed77b62cbab66af70f6baf1179f7dceb"}
Oct 31 01:52:09.085076 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:09.084996    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft"
Oct 31 01:52:09.103994 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:09.103943    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf" podStartSLOduration=2.103925147 podStartE2EDuration="2.103925147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:09.103869621 +0000 UTC m=+1885.107583928" watchObservedRunningTime="2026-10-31 01:52:09.103925147 +0000 UTC m=+1885.107639438"
Oct 31 01:52:09.130220 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:09.130154    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft" podStartSLOduration=2.130139008 podStartE2EDuration="2.130139008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:09.128757022 +0000 UTC m=+1885.132471315" watchObservedRunningTime="2026-10-31 01:52:09.130139008 +0000 UTC m=+1885.133853298"
Oct 31 01:52:09.393258 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:09.393225    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7"
Oct 31 01:52:09.397797 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:09.397772    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs"
Oct 31 01:52:09.492796 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:09.492741    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ab4857-a9ba-4524-9025-d2906539f9c3-kube-api-access-r5qrh\" (UniqueName: \"kubernetes.io/projected/c0ab4857-a9ba-4524-9025-d2906539f9c3-kube-api-access-r5qrh\") pod \"c0ab4857-a9ba-4524-9025-d2906539f9c3\" (UID: \"c0ab4857-a9ba-4524-9025-d2906539f9c3\") "
Oct 31 01:52:09.492796 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:09.492794    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14a414fd-b21c-4086-bae8-090b9c22ba38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a414fd-b21c-4086-bae8-090b9c22ba38-plugins-config-volume\") pod \"14a414fd-b21c-4086-bae8-090b9c22ba38\" (UID: \"14a414fd-b21c-4086-bae8-090b9c22ba38\") "
Oct 31 01:52:09.492935 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:09.492830    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ab4857-a9ba-4524-9025-d2906539f9c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ab4857-a9ba-4524-9025-d2906539f9c3-plugins-config-volume\") pod \"c0ab4857-a9ba-4524-9025-d2906539f9c3\" (UID: \"c0ab4857-a9ba-4524-9025-d2906539f9c3\") "
Oct 31 01:52:09.492978 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:09.492959    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a414fd-b21c-4086-bae8-090b9c22ba38-kube-api-access-4nksr\" (UniqueName: \"kubernetes.io/projected/14a414fd-b21c-4086-bae8-090b9c22ba38-kube-api-access-4nksr\") pod \"14a414fd-b21c-4086-bae8-090b9c22ba38\" (UID: \"14a414fd-b21c-4086-bae8-090b9c22ba38\") "
Oct 31 01:52:09.493269 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:09.493244    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a414fd-b21c-4086-bae8-090b9c22ba38-plugins-config-volume" pod "14a414fd-b21c-4086-bae8-090b9c22ba38" (UID: "14a414fd-b21c-4086-bae8-090b9c22ba38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:09.493408 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:09.493295    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ab4857-a9ba-4524-9025-d2906539f9c3-plugins-config-volume" pod "c0ab4857-a9ba-4524-9025-d2906539f9c3" (UID: "c0ab4857-a9ba-4524-9025-d2906539f9c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:09.493613 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:09.493596    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14a414fd-b21c-4086-bae8-090b9c22ba38-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:09.493675 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:09.493615    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ab4857-a9ba-4524-9025-d2906539f9c3-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:09.494752 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:09.494729    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ab4857-a9ba-4524-9025-d2906539f9c3-kube-api-access-r5qrh" pod "c0ab4857-a9ba-4524-9025-d2906539f9c3" (UID: "c0ab4857-a9ba-4524-9025-d2906539f9c3"). InnerVolumeSpecName "kube-api-access-r5qrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:09.494965 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:09.494947    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a414fd-b21c-4086-bae8-090b9c22ba38-kube-api-access-4nksr" pod "14a414fd-b21c-4086-bae8-090b9c22ba38" (UID: "14a414fd-b21c-4086-bae8-090b9c22ba38"). InnerVolumeSpecName "kube-api-access-4nksr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:09.594652 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:09.594566    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nksr\" (UniqueName: \"kubernetes.io/projected/14a414fd-b21c-4086-bae8-090b9c22ba38-kube-api-access-4nksr\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:09.594652 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:09.594595    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5qrh\" (UniqueName: \"kubernetes.io/projected/c0ab4857-a9ba-4524-9025-d2906539f9c3-kube-api-access-r5qrh\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:10.093349 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:10.093311    5650 generic.go:356] "Generic (PLEG): container finished" podID="c0ab4857-a9ba-4524-9025-d2906539f9c3" containerID="7219246d9bb388351b0d5d7cdb622128a1c1b680cf3ac00f0551b6bace33a7ad" exitCode=0
Oct 31 01:52:10.093855 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:10.093371    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7"
Oct 31 01:52:10.093855 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:10.093399    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7" event={"ID":"c0ab4857-a9ba-4524-9025-d2906539f9c3","Type":"ContainerDied","Data":"7219246d9bb388351b0d5d7cdb622128a1c1b680cf3ac00f0551b6bace33a7ad"}
Oct 31 01:52:10.093855 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:10.093526    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7" event={"ID":"c0ab4857-a9ba-4524-9025-d2906539f9c3","Type":"ContainerDied","Data":"41fcaf41650b4e61b7169c08ec217f1fa363820ba463453c146fb45a5d405568"}
Oct 31 01:52:10.093855 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:10.093556    5650 scope.go:122] "RemoveContainer" containerID="7219246d9bb388351b0d5d7cdb622128a1c1b680cf3ac00f0551b6bace33a7ad"
Oct 31 01:52:10.095561 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:10.095536    5650 generic.go:356] "Generic (PLEG): container finished" podID="eed0a2f2-6812-4d37-b175-7ca1d14d5f9c" containerID="af785e4b02c3d66b8906969382a903d60fe512552ec3a80ef4bf3e2667ec82e8" exitCode=0
Oct 31 01:52:10.095680 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:10.095613    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq" event={"ID":"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c","Type":"ContainerDied","Data":"af785e4b02c3d66b8906969382a903d60fe512552ec3a80ef4bf3e2667ec82e8"}
Oct 31 01:52:10.096103 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:10.096085    5650 scope.go:122] "RemoveContainer" containerID="af785e4b02c3d66b8906969382a903d60fe512552ec3a80ef4bf3e2667ec82e8"
Oct 31 01:52:10.097146 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:10.097102    5650 generic.go:356] "Generic (PLEG): container finished" podID="14a414fd-b21c-4086-bae8-090b9c22ba38" containerID="747f8d17c5dc8f293b5277bebae773ffc92deb039338fbf9bc5ae6fb1d588962" exitCode=0
Oct 31 01:52:10.097239 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:10.097191    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs"
Oct 31 01:52:10.097278 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:10.097262    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs" event={"ID":"14a414fd-b21c-4086-bae8-090b9c22ba38","Type":"ContainerDied","Data":"747f8d17c5dc8f293b5277bebae773ffc92deb039338fbf9bc5ae6fb1d588962"}
Oct 31 01:52:10.097315 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:10.097304    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs" event={"ID":"14a414fd-b21c-4086-bae8-090b9c22ba38","Type":"ContainerDied","Data":"c41c93136ebcc4baac7a328991a7a9d39732220ea23bb80e100af05ecbd93060"}
Oct 31 01:52:10.098052 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:10.098019    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:52:10.098136 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:10.098112    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:52:10.098175 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:10.098165    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:52:10.105516 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:10.105431    5650 scope.go:122] "RemoveContainer" containerID="7219246d9bb388351b0d5d7cdb622128a1c1b680cf3ac00f0551b6bace33a7ad"
Oct 31 01:52:10.105848 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:52:10.105819    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7219246d9bb388351b0d5d7cdb622128a1c1b680cf3ac00f0551b6bace33a7ad\": container with ID starting with 7219246d9bb388351b0d5d7cdb622128a1c1b680cf3ac00f0551b6bace33a7ad not found: ID does not exist" containerID="7219246d9bb388351b0d5d7cdb622128a1c1b680cf3ac00f0551b6bace33a7ad"
Oct 31 01:52:10.105950 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:10.105850    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7219246d9bb388351b0d5d7cdb622128a1c1b680cf3ac00f0551b6bace33a7ad"} err="failed to get container status \"7219246d9bb388351b0d5d7cdb622128a1c1b680cf3ac00f0551b6bace33a7ad\": rpc error: code = NotFound desc = could not find container \"7219246d9bb388351b0d5d7cdb622128a1c1b680cf3ac00f0551b6bace33a7ad\": container with ID starting with 7219246d9bb388351b0d5d7cdb622128a1c1b680cf3ac00f0551b6bace33a7ad not found: ID does not exist"
Oct 31 01:52:10.105950 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:10.105866    5650 scope.go:122] "RemoveContainer" containerID="747f8d17c5dc8f293b5277bebae773ffc92deb039338fbf9bc5ae6fb1d588962"
Oct 31 01:52:10.117589 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:10.117568    5650 scope.go:122] "RemoveContainer" containerID="747f8d17c5dc8f293b5277bebae773ffc92deb039338fbf9bc5ae6fb1d588962"
Oct 31 01:52:10.118206 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:52:10.118183    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"747f8d17c5dc8f293b5277bebae773ffc92deb039338fbf9bc5ae6fb1d588962\": container with ID starting with 747f8d17c5dc8f293b5277bebae773ffc92deb039338fbf9bc5ae6fb1d588962 not found: ID does not exist" containerID="747f8d17c5dc8f293b5277bebae773ffc92deb039338fbf9bc5ae6fb1d588962"
Oct 31 01:52:10.118313 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:10.118212    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"747f8d17c5dc8f293b5277bebae773ffc92deb039338fbf9bc5ae6fb1d588962"} err="failed to get container status \"747f8d17c5dc8f293b5277bebae773ffc92deb039338fbf9bc5ae6fb1d588962\": rpc error: code = NotFound desc = could not find container \"747f8d17c5dc8f293b5277bebae773ffc92deb039338fbf9bc5ae6fb1d588962\": container with ID starting with 747f8d17c5dc8f293b5277bebae773ffc92deb039338fbf9bc5ae6fb1d588962 not found: ID does not exist"
Oct 31 01:52:10.161392 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:10.161361    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs"]
Oct 31 01:52:10.172508 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:10.172480    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c06ae4-6c7b457f5-nw9bs"]
Oct 31 01:52:10.189650 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:10.189609    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7"]
Oct 31 01:52:10.194637 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:10.194609    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c06ae4-85bbdc8b95-2vgf7"]
Oct 31 01:52:11.122470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:11.117473    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq" event={"ID":"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c","Type":"ContainerStarted","Data":"fcf4c4dcc2fd7b674b3ee7c0e2ad9e14b8283c0211760399130dbcbf0285427b"}
Oct 31 01:52:11.722159 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:11.722100    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"]
Oct 31 01:52:11.722769 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:11.722708    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t" podUID="fec61f6a-7103-49ef-ae2c-bbe826dcd159" containerName="istio-proxy" containerID="cri-o://9e35bf6df1a1ed5a89453788b676949f6836d64ce36d0d0ee40c15ecffb44d20" gracePeriod=30
Oct 31 01:52:12.072596 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:12.072512    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:52:12.073888 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:12.073865    5650 patch_prober.go:29] interesting pod/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 31 01:52:12.073951 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:12.073913    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq" podUID="eed0a2f2-6812-4d37-b175-7ca1d14d5f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 31 01:52:12.137361 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:12.137328    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14a414fd-b21c-4086-bae8-090b9c22ba38" path="/var/lib/kubelet/pods/14a414fd-b21c-4086-bae8-090b9c22ba38/volumes"
Oct 31 01:52:12.138030 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:12.138008    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0ab4857-a9ba-4524-9025-d2906539f9c3" path="/var/lib/kubelet/pods/c0ab4857-a9ba-4524-9025-d2906539f9c3/volumes"
Oct 31 01:52:13.030285 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:13.030122    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-271413cd-q9q2c"]
Oct 31 01:52:13.034165 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:13.034133    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-271413cd-q9q2c"]
Oct 31 01:52:13.073084 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:13.073041    5650 patch_prober.go:29] interesting pod/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 31 01:52:13.073297 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:13.073130    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq" podUID="eed0a2f2-6812-4d37-b175-7ca1d14d5f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 31 01:52:14.072861 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:14.072820    5650 patch_prober.go:29] interesting pod/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 31 01:52:14.073279 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:14.072882    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq" podUID="eed0a2f2-6812-4d37-b175-7ca1d14d5f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 31 01:52:14.139645 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:14.139607    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d94739d-327b-4ece-9b0a-bf38caeae06e" path="/var/lib/kubelet/pods/3d94739d-327b-4ece-9b0a-bf38caeae06e/volumes"
Oct 31 01:52:15.072686 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:15.072649    5650 patch_prober.go:29] interesting pod/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 31 01:52:15.072889 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:15.072702    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq" podUID="eed0a2f2-6812-4d37-b175-7ca1d14d5f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 31 01:52:15.849856 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:15.849819    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:52:16.072763 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:16.072726    5650 patch_prober.go:29] interesting pod/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 31 01:52:16.072961 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:16.072788    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq" podUID="eed0a2f2-6812-4d37-b175-7ca1d14d5f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 31 01:52:17.032495 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.032469    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:52:17.078675 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.078636    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:52:17.079964 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.079940    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:52:17.155192 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.155151    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-workload-certs\") pod \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") "
Oct 31 01:52:17.155466 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.155430    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istiod-ca-cert\") pod \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") "
Oct 31 01:52:17.155935 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.155889    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-credential-socket\") pod \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") "
Oct 31 01:52:17.156305 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.156287    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-data\") pod \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") "
Oct 31 01:52:17.156540 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.156524    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-podinfo\") pod \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") "
Oct 31 01:52:17.156771 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.156757    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-workload-socket\") pod \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") "
Oct 31 01:52:17.156943 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.156921    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-token\" (UniqueName: \"kubernetes.io/projected/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-token\") pod \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") "
Oct 31 01:52:17.157081 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.156967    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-envoy\") pod \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") "
Oct 31 01:52:17.157144 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.157087    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fec61f6a-7103-49ef-ae2c-bbe826dcd159-kube-api-access-z8pdh\" (UniqueName: \"kubernetes.io/projected/fec61f6a-7103-49ef-ae2c-bbe826dcd159-kube-api-access-z8pdh\") pod \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\" (UID: \"fec61f6a-7103-49ef-ae2c-bbe826dcd159\") "
Oct 31 01:52:17.160763 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.160735    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-workload-socket" pod "fec61f6a-7103-49ef-ae2c-bbe826dcd159" (UID: "fec61f6a-7103-49ef-ae2c-bbe826dcd159"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:17.161627 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.161598    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-workload-certs" pod "fec61f6a-7103-49ef-ae2c-bbe826dcd159" (UID: "fec61f6a-7103-49ef-ae2c-bbe826dcd159"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:17.162163 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.162139    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-data" pod "fec61f6a-7103-49ef-ae2c-bbe826dcd159" (UID: "fec61f6a-7103-49ef-ae2c-bbe826dcd159"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:17.162383 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.162361    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-credential-socket" pod "fec61f6a-7103-49ef-ae2c-bbe826dcd159" (UID: "fec61f6a-7103-49ef-ae2c-bbe826dcd159"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:17.163694 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.163669    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istiod-ca-cert" pod "fec61f6a-7103-49ef-ae2c-bbe826dcd159" (UID: "fec61f6a-7103-49ef-ae2c-bbe826dcd159"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:17.168063 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.168036    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-token" pod "fec61f6a-7103-49ef-ae2c-bbe826dcd159" (UID: "fec61f6a-7103-49ef-ae2c-bbe826dcd159"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:17.170369 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.170332    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-podinfo" pod "fec61f6a-7103-49ef-ae2c-bbe826dcd159" (UID: "fec61f6a-7103-49ef-ae2c-bbe826dcd159"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:52:17.172921 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.172836    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-envoy" pod "fec61f6a-7103-49ef-ae2c-bbe826dcd159" (UID: "fec61f6a-7103-49ef-ae2c-bbe826dcd159"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:17.174965 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.174926    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec61f6a-7103-49ef-ae2c-bbe826dcd159-kube-api-access-z8pdh" pod "fec61f6a-7103-49ef-ae2c-bbe826dcd159" (UID: "fec61f6a-7103-49ef-ae2c-bbe826dcd159"). InnerVolumeSpecName "kube-api-access-z8pdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:17.187970 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.187864    5650 generic.go:356] "Generic (PLEG): container finished" podID="fec61f6a-7103-49ef-ae2c-bbe826dcd159" containerID="9e35bf6df1a1ed5a89453788b676949f6836d64ce36d0d0ee40c15ecffb44d20" exitCode=0
Oct 31 01:52:17.188128 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.187925    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t" event={"ID":"fec61f6a-7103-49ef-ae2c-bbe826dcd159","Type":"ContainerDied","Data":"9e35bf6df1a1ed5a89453788b676949f6836d64ce36d0d0ee40c15ecffb44d20"}
Oct 31 01:52:17.189334 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.187950    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"
Oct 31 01:52:17.204698 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.199678    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t" event={"ID":"fec61f6a-7103-49ef-ae2c-bbe826dcd159","Type":"ContainerDied","Data":"f471e525229cfcf6af1e5c3b1a80d7e1ccb3f3f66adb39705b6fb34211c57876"}
Oct 31 01:52:17.204698 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.199734    5650 scope.go:122] "RemoveContainer" containerID="9e35bf6df1a1ed5a89453788b676949f6836d64ce36d0d0ee40c15ecffb44d20"
Oct 31 01:52:17.230792 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.230570    5650 scope.go:122] "RemoveContainer" containerID="b279e1bd517d054265149f6052141ba7260ab6f000c754a18f21c3aa903f9f2c"
Oct 31 01:52:17.249097 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.249024    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"]
Oct 31 01:52:17.253825 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.253789    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-c06ae4-openshift-default-5448bdf45c-fqw8t"]
Oct 31 01:52:17.259599 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.259574    5650 scope.go:122] "RemoveContainer" containerID="9e35bf6df1a1ed5a89453788b676949f6836d64ce36d0d0ee40c15ecffb44d20"
Oct 31 01:52:17.259962 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:52:17.259937    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e35bf6df1a1ed5a89453788b676949f6836d64ce36d0d0ee40c15ecffb44d20\": container with ID starting with 9e35bf6df1a1ed5a89453788b676949f6836d64ce36d0d0ee40c15ecffb44d20 not found: ID does not exist" containerID="9e35bf6df1a1ed5a89453788b676949f6836d64ce36d0d0ee40c15ecffb44d20"
Oct 31 01:52:17.260039 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.259976    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e35bf6df1a1ed5a89453788b676949f6836d64ce36d0d0ee40c15ecffb44d20"} err="failed to get container status \"9e35bf6df1a1ed5a89453788b676949f6836d64ce36d0d0ee40c15ecffb44d20\": rpc error: code = NotFound desc = could not find container \"9e35bf6df1a1ed5a89453788b676949f6836d64ce36d0d0ee40c15ecffb44d20\": container with ID starting with 9e35bf6df1a1ed5a89453788b676949f6836d64ce36d0d0ee40c15ecffb44d20 not found: ID does not exist"
Oct 31 01:52:17.260039 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.260005    5650 scope.go:122] "RemoveContainer" containerID="b279e1bd517d054265149f6052141ba7260ab6f000c754a18f21c3aa903f9f2c"
Oct 31 01:52:17.260334 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:52:17.260310    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b279e1bd517d054265149f6052141ba7260ab6f000c754a18f21c3aa903f9f2c\": container with ID starting with b279e1bd517d054265149f6052141ba7260ab6f000c754a18f21c3aa903f9f2c not found: ID does not exist" containerID="b279e1bd517d054265149f6052141ba7260ab6f000c754a18f21c3aa903f9f2c"
Oct 31 01:52:17.260437 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.260340    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b279e1bd517d054265149f6052141ba7260ab6f000c754a18f21c3aa903f9f2c"} err="failed to get container status \"b279e1bd517d054265149f6052141ba7260ab6f000c754a18f21c3aa903f9f2c\": rpc error: code = NotFound desc = could not find container \"b279e1bd517d054265149f6052141ba7260ab6f000c754a18f21c3aa903f9f2c\": container with ID starting with b279e1bd517d054265149f6052141ba7260ab6f000c754a18f21c3aa903f9f2c not found: ID does not exist"
Oct 31 01:52:17.261226 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.261208    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8pdh\" (UniqueName: \"kubernetes.io/projected/fec61f6a-7103-49ef-ae2c-bbe826dcd159-kube-api-access-z8pdh\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:17.261275 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.261230    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:17.261275 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.261242    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:17.261275 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.261255    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:17.261275 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.261267    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:17.261420 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.261280    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:17.261420 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.261291    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:17.261420 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.261300    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:17.261420 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:17.261309    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fec61f6a-7103-49ef-ae2c-bbe826dcd159-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:18.119862 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:18.119823    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fec61f6a-7103-49ef-ae2c-bbe826dcd159" path="/var/lib/kubelet/pods/fec61f6a-7103-49ef-ae2c-bbe826dcd159/volumes"
Oct 31 01:52:20.101393 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:20.101363    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf"
Oct 31 01:52:20.101905 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:20.101435    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft"
Oct 31 01:52:24.997731 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:24.997644    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7"]
Oct 31 01:52:24.998329 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:24.998312    5650 cpu_manager.go:414] "containerMap: removing container" podUID="c0ab4857-a9ba-4524-9025-d2906539f9c3" containerName="payload-processing"
Oct 31 01:52:24.998373 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:24.998331    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ab4857-a9ba-4524-9025-d2906539f9c3" containerName="payload-processing"
Oct 31 01:52:24.998409 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:24.998374    5650 cpu_manager.go:414] "containerMap: removing container" podUID="fec61f6a-7103-49ef-ae2c-bbe826dcd159" containerName="istio-proxy"
Oct 31 01:52:24.998409 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:24.998379    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec61f6a-7103-49ef-ae2c-bbe826dcd159" containerName="istio-proxy"
Oct 31 01:52:24.998409 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:24.998396    5650 cpu_manager.go:414] "containerMap: removing container" podUID="14a414fd-b21c-4086-bae8-090b9c22ba38" containerName="payload-pre-processing"
Oct 31 01:52:24.998409 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:24.998401    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a414fd-b21c-4086-bae8-090b9c22ba38" containerName="payload-pre-processing"
Oct 31 01:52:24.998547 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:24.998513    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec61f6a-7103-49ef-ae2c-bbe826dcd159" containerName="istio-proxy"
Oct 31 01:52:24.998547 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:24.998524    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="14a414fd-b21c-4086-bae8-090b9c22ba38" containerName="payload-pre-processing"
Oct 31 01:52:24.998547 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:24.998533    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ab4857-a9ba-4524-9025-d2906539f9c3" containerName="payload-processing"
Oct 31 01:52:25.003613 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.003592    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7"
Oct 31 01:52:25.010135 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.010109    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7"]
Oct 31 01:52:25.011047 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.011020    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-925bac\""
Oct 31 01:52:25.037523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.037484    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6"]
Oct 31 01:52:25.038744 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.038472    5650 cpu_manager.go:414] "containerMap: removing container" podUID="fec61f6a-7103-49ef-ae2c-bbe826dcd159" containerName="istio-proxy"
Oct 31 01:52:25.038744 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.038496    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec61f6a-7103-49ef-ae2c-bbe826dcd159" containerName="istio-proxy"
Oct 31 01:52:25.038943 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.038821    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec61f6a-7103-49ef-ae2c-bbe826dcd159" containerName="istio-proxy"
Oct 31 01:52:25.045713 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.045687    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6"]
Oct 31 01:52:25.046126 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.045826    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6"
Oct 31 01:52:25.055581 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.053711    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-925bac-dockercfg-q26qw\""
Oct 31 01:52:25.093405 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.093368    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57c2b\" (UniqueName: \"kubernetes.io/projected/840dad17-e4e9-4a19-83d4-af8ce9ea1aad-kube-api-access-57c2b\") pod \"payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6\" (UID: \"840dad17-e4e9-4a19-83d4-af8ce9ea1aad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6"
Oct 31 01:52:25.093600 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.093548    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe46844f-a0d6-4210-8516-f4facfbe165f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7\" (UID: \"fe46844f-a0d6-4210-8516-f4facfbe165f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7"
Oct 31 01:52:25.093600 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.093592    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/840dad17-e4e9-4a19-83d4-af8ce9ea1aad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6\" (UID: \"840dad17-e4e9-4a19-83d4-af8ce9ea1aad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6"
Oct 31 01:52:25.093731 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.093715    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ls6t\" (UniqueName: \"kubernetes.io/projected/fe46844f-a0d6-4210-8516-f4facfbe165f-kube-api-access-6ls6t\") pod \"payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7\" (UID: \"fe46844f-a0d6-4210-8516-f4facfbe165f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7"
Oct 31 01:52:25.205468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.194875    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57c2b\" (UniqueName: \"kubernetes.io/projected/840dad17-e4e9-4a19-83d4-af8ce9ea1aad-kube-api-access-57c2b\") pod \"payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6\" (UID: \"840dad17-e4e9-4a19-83d4-af8ce9ea1aad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6"
Oct 31 01:52:25.205468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.195420    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe46844f-a0d6-4210-8516-f4facfbe165f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7\" (UID: \"fe46844f-a0d6-4210-8516-f4facfbe165f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7"
Oct 31 01:52:25.216489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.209934    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe46844f-a0d6-4210-8516-f4facfbe165f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7\" (UID: \"fe46844f-a0d6-4210-8516-f4facfbe165f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7"
Oct 31 01:52:25.216489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.213235    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/840dad17-e4e9-4a19-83d4-af8ce9ea1aad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6\" (UID: \"840dad17-e4e9-4a19-83d4-af8ce9ea1aad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6"
Oct 31 01:52:25.216489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.213368    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ls6t\" (UniqueName: \"kubernetes.io/projected/fe46844f-a0d6-4210-8516-f4facfbe165f-kube-api-access-6ls6t\") pod \"payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7\" (UID: \"fe46844f-a0d6-4210-8516-f4facfbe165f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7"
Oct 31 01:52:25.237473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.235686    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ls6t\" (UniqueName: \"kubernetes.io/projected/fe46844f-a0d6-4210-8516-f4facfbe165f-kube-api-access-6ls6t\") pod \"payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7\" (UID: \"fe46844f-a0d6-4210-8516-f4facfbe165f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7"
Oct 31 01:52:25.242469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.239361    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57c2b\" (UniqueName: \"kubernetes.io/projected/840dad17-e4e9-4a19-83d4-af8ce9ea1aad-kube-api-access-57c2b\") pod \"payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6\" (UID: \"840dad17-e4e9-4a19-83d4-af8ce9ea1aad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6"
Oct 31 01:52:25.242469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.240066    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/840dad17-e4e9-4a19-83d4-af8ce9ea1aad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6\" (UID: \"840dad17-e4e9-4a19-83d4-af8ce9ea1aad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6"
Oct 31 01:52:25.321088 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.319861    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7"
Oct 31 01:52:25.369337 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.369306    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6"
Oct 31 01:52:25.688669 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:52:25.688584    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe46844f_a0d6_4210_8516_f4facfbe165f.slice/crio-f18b04f9128ec98e1a764b8f2303fef06a69532a3f1cb540540029f48896f67c WatchSource:0}: Error finding container f18b04f9128ec98e1a764b8f2303fef06a69532a3f1cb540540029f48896f67c: Status 404 returned error can't find the container with id f18b04f9128ec98e1a764b8f2303fef06a69532a3f1cb540540029f48896f67c
Oct 31 01:52:25.689016 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.688989    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7"]
Oct 31 01:52:25.704565 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:52:25.699590    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod840dad17_e4e9_4a19_83d4_af8ce9ea1aad.slice/crio-74978ad36a4c5f811b2b1871e584e2ff2ac4dbacb8061f862bfa9a4ff619c78a WatchSource:0}: Error finding container 74978ad36a4c5f811b2b1871e584e2ff2ac4dbacb8061f862bfa9a4ff619c78a: Status 404 returned error can't find the container with id 74978ad36a4c5f811b2b1871e584e2ff2ac4dbacb8061f862bfa9a4ff619c78a
Oct 31 01:52:25.704565 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:25.702596    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6"]
Oct 31 01:52:26.296930 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.296886    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7" event={"ID":"fe46844f-a0d6-4210-8516-f4facfbe165f","Type":"ContainerStarted","Data":"483d7abc0bb3d588e2fa0530e76682ab6027a3a530ca8ddcb07bf175ef9c7d09"}
Oct 31 01:52:26.297486 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.296956    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7" event={"ID":"fe46844f-a0d6-4210-8516-f4facfbe165f","Type":"ContainerStarted","Data":"f18b04f9128ec98e1a764b8f2303fef06a69532a3f1cb540540029f48896f67c"}
Oct 31 01:52:26.297486 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.296990    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7"
Oct 31 01:52:26.298824 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.298793    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6" event={"ID":"840dad17-e4e9-4a19-83d4-af8ce9ea1aad","Type":"ContainerStarted","Data":"3fad9e812ad79c439393ed5082cb63e12852c9b801c8d09ba3b1501851340652"}
Oct 31 01:52:26.298953 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.298849    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6" event={"ID":"840dad17-e4e9-4a19-83d4-af8ce9ea1aad","Type":"ContainerStarted","Data":"74978ad36a4c5f811b2b1871e584e2ff2ac4dbacb8061f862bfa9a4ff619c78a"}
Oct 31 01:52:26.298953 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.298880    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6"
Oct 31 01:52:26.323029 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.322975    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7" podStartSLOduration=2.32296172 podStartE2EDuration="2.32296172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:26.317518545 +0000 UTC m=+1902.321232836" watchObservedRunningTime="2026-10-31 01:52:26.32296172 +0000 UTC m=+1902.326676010"
Oct 31 01:52:26.344826 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.344761    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6" podStartSLOduration=1.344742189 podStartE2EDuration="1.344742189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:26.340919354 +0000 UTC m=+1902.344633646" watchObservedRunningTime="2026-10-31 01:52:26.344742189 +0000 UTC m=+1902.348456484"
Oct 31 01:52:26.629339 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.629244    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"]
Oct 31 01:52:26.647800 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.646398    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.647800 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.646663    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"]
Oct 31 01:52:26.660469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.658744    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-a779b3-openshift-default-dockercfg-nkdjw\""
Oct 31 01:52:26.751803 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.751765    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glswb\" (UniqueName: \"kubernetes.io/projected/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-kube-api-access-glswb\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.751803 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.751806    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-credential-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.752060 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.751954    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istiod-ca-cert\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.752060 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.752009    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-podinfo\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.752176 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.752131    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-workload-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.752176 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.752165    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-data\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.752279 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.752200    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-workload-certs\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.752279 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.752250    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-envoy\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.752381 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.752296    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-token\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.853627 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.853582    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-token\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.853818 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.853796    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glswb\" (UniqueName: \"kubernetes.io/projected/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-kube-api-access-glswb\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.853864 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.853839    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-credential-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.853934 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.853917    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istiod-ca-cert\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.853976 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.853960    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-podinfo\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.854109 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.854090    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-workload-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.854218 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.854139    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-data\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.854218 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.854179    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-workload-certs\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.854218 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.854214    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-envoy\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.854380 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.854291    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-credential-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.854522 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.854497    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-workload-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.854603 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.854577    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-data\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.854751 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.854730    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istiod-ca-cert\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.854818 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.854772    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-workload-certs\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.856729 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.856705    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-envoy\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.856833 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.856788    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-podinfo\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.866813 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.866783    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-token\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.866953 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.866875    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glswb\" (UniqueName: \"kubernetes.io/projected/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-kube-api-access-glswb\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:26.986388 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:26.978513    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:27.454529 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:27.452638    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"]
Oct 31 01:52:27.540216 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:27.540163    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:52:27.540395 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:27.540274    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:52:27.540395 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:27.540324    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:52:28.779468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:28.766376    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd" event={"ID":"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c","Type":"ContainerStarted","Data":"396de444acf838c03c82bfca70c53ce0a870ed2a8fa9b81a90a1338f413700c0"}
Oct 31 01:52:28.779468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:28.766841    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd" event={"ID":"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c","Type":"ContainerStarted","Data":"2584a7802f015b39b7a5553b44de782fa2973b64125a5db68c59a60ee3cedda1"}
Oct 31 01:52:28.876471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:28.876279    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd" podStartSLOduration=2.876259426 podStartE2EDuration="2.876259426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:28.868413677 +0000 UTC m=+1904.872127971" watchObservedRunningTime="2026-10-31 01:52:28.876259426 +0000 UTC m=+1904.879973719"
Oct 31 01:52:28.971063 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:52:28.970643    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:52:28.994479 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:28.984777    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:28.994479 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:28.984981    5650 patch_prober.go:29] interesting pod/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 31 01:52:28.994479 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:28.985034    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd" podUID="69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 31 01:52:29.979474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:29.979334    5650 patch_prober.go:29] interesting pod/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 31 01:52:29.979474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:29.979398    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd" podUID="69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 31 01:52:30.979951 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:30.979914    5650 patch_prober.go:29] interesting pod/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 31 01:52:30.980481 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:30.979979    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd" podUID="69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 31 01:52:31.979568 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:31.979534    5650 patch_prober.go:29] interesting pod/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 31 01:52:31.979743 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:31.979588    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd" podUID="69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 31 01:52:32.979029 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:32.978987    5650 patch_prober.go:29] interesting pod/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 31 01:52:32.979424 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:32.979058    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd" podUID="69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 31 01:52:33.979348 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:33.979309    5650 patch_prober.go:29] interesting pod/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 31 01:52:33.979849 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:33.979379    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd" podUID="69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 31 01:52:34.985061 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:34.985027    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:34.985556 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:34.985433    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:34.986317 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:34.986297    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:52:37.129362 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.129326    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"]
Oct 31 01:52:37.133670 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.133642    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.150530 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.150498    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-b8d890-openshift-default-dockercfg-fvs7g\""
Oct 31 01:52:37.153086 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.153056    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"]
Oct 31 01:52:37.235791 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.235753    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-istio-envoy\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.235960 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.235807    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-istio-data\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.235960 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.235948    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-workload-socket\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.236060 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.235972    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb9db005-6073-4c28-bec1-36495261155c-istio-token\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.236060 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.236009    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb9db005-6073-4c28-bec1-36495261155c-istiod-ca-cert\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.236167 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.236154    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-workload-certs\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.236224 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.236173    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb9db005-6073-4c28-bec1-36495261155c-istio-podinfo\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.236224 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.236219    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-credential-socket\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.236343 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.236258    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x86tp\" (UniqueName: \"kubernetes.io/projected/cb9db005-6073-4c28-bec1-36495261155c-kube-api-access-x86tp\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.275871 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.275824    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"]
Oct 31 01:52:37.276690 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.276338    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp" podUID="35cccf52-b489-42d0-ae01-f3071bdc00ed" containerName="istio-proxy" containerID="cri-o://75e24e03ff8c864169d6bebdd6ef912ef5e79bc626feea5471a72e6b2c1814cd" gracePeriod=30
Oct 31 01:52:37.337275 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.337239    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-istio-envoy\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.337455 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.337287    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-istio-data\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.337455 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.337358    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-workload-socket\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.337455 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.337382    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb9db005-6073-4c28-bec1-36495261155c-istio-token\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.337455 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.337420    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb9db005-6073-4c28-bec1-36495261155c-istiod-ca-cert\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.337693 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.337524    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-workload-certs\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.337693 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.337549    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb9db005-6073-4c28-bec1-36495261155c-istio-podinfo\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.337693 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.337597    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-credential-socket\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.337693 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.337654    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x86tp\" (UniqueName: \"kubernetes.io/projected/cb9db005-6073-4c28-bec1-36495261155c-kube-api-access-x86tp\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.337916 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.337844    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-workload-socket\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.338163 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.338130    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-workload-certs\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.338343 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.338231    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb9db005-6073-4c28-bec1-36495261155c-istiod-ca-cert\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.339006 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.338978    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-credential-socket\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.339724 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.339699    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-istio-data\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.340421 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.340392    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-istio-envoy\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.340764 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.340744    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb9db005-6073-4c28-bec1-36495261155c-istio-podinfo\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.355649 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.353661    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb9db005-6073-4c28-bec1-36495261155c-istio-token\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.355649 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.353774    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x86tp\" (UniqueName: \"kubernetes.io/projected/cb9db005-6073-4c28-bec1-36495261155c-kube-api-access-x86tp\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.454093 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.453992    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:37.505693 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.505661    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6"
Oct 31 01:52:37.507770 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.507716    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7"
Oct 31 01:52:37.821937 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.821903    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"]
Oct 31 01:52:37.822065 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:52:37.821974    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9db005_6073_4c28_bec1_36495261155c.slice/crio-59063a370c3a0211500958a1cc4808b61cd87cd1df0ce4254262745b62cf91c6 WatchSource:0}: Error finding container 59063a370c3a0211500958a1cc4808b61cd87cd1df0ce4254262745b62cf91c6: Status 404 returned error can't find the container with id 59063a370c3a0211500958a1cc4808b61cd87cd1df0ce4254262745b62cf91c6
Oct 31 01:52:37.825261 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.825227    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:52:37.825334 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.825296    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:52:37.825334 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.825325    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:52:37.879613 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:37.879580    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p" event={"ID":"cb9db005-6073-4c28-bec1-36495261155c","Type":"ContainerStarted","Data":"59063a370c3a0211500958a1cc4808b61cd87cd1df0ce4254262745b62cf91c6"}
Oct 31 01:52:38.400323 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:38.400286    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t"]
Oct 31 01:52:38.418364 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:38.418330    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp"]
Oct 31 01:52:38.929054 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:38.929017    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p" event={"ID":"cb9db005-6073-4c28-bec1-36495261155c","Type":"ContainerStarted","Data":"745fced1f9af15956a1344fbb6497548dc53fc84d6218f4936d361d917615e5b"}
Oct 31 01:52:38.958275 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:38.958222    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p" podStartSLOduration=1.9582073819999999 podStartE2EDuration="1.958207382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:38.955351944 +0000 UTC m=+1914.959066235" watchObservedRunningTime="2026-10-31 01:52:38.958207382 +0000 UTC m=+1914.961921650"
Oct 31 01:52:39.454767 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:39.454731    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:39.456310 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:39.456277    5650 patch_prober.go:29] interesting pod/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 31 01:52:39.456433 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:39.456328    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p" podUID="cb9db005-6073-4c28-bec1-36495261155c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 31 01:52:40.456486 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:40.454673    5650 patch_prober.go:29] interesting pod/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 31 01:52:40.456486 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:40.454756    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p" podUID="cb9db005-6073-4c28-bec1-36495261155c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 31 01:52:41.456405 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:41.456365    5650 patch_prober.go:29] interesting pod/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 31 01:52:41.456786 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:41.456750    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p" podUID="cb9db005-6073-4c28-bec1-36495261155c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 31 01:52:42.454632 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.454593    5650 patch_prober.go:29] interesting pod/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 31 01:52:42.454787 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.454653    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p" podUID="cb9db005-6073-4c28-bec1-36495261155c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 31 01:52:42.678180 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.678150    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:52:42.744131 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.743672    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35cccf52-b489-42d0-ae01-f3071bdc00ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cccf52-b489-42d0-ae01-f3071bdc00ed-istiod-ca-cert\") pod \"35cccf52-b489-42d0-ae01-f3071bdc00ed\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") "
Oct 31 01:52:42.744580 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.744545    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-envoy\") pod \"35cccf52-b489-42d0-ae01-f3071bdc00ed\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") "
Oct 31 01:52:42.745189 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.744087    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35cccf52-b489-42d0-ae01-f3071bdc00ed-istiod-ca-cert" pod "35cccf52-b489-42d0-ae01-f3071bdc00ed" (UID: "35cccf52-b489-42d0-ae01-f3071bdc00ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:42.745632 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.745596    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-data" pod "35cccf52-b489-42d0-ae01-f3071bdc00ed" (UID: "35cccf52-b489-42d0-ae01-f3071bdc00ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:42.745763 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.745746    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-data\") pod \"35cccf52-b489-42d0-ae01-f3071bdc00ed\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") "
Oct 31 01:52:42.746030 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.746012    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-token\" (UniqueName: \"kubernetes.io/projected/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-token\") pod \"35cccf52-b489-42d0-ae01-f3071bdc00ed\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") "
Oct 31 01:52:42.746532 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.746514    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-workload-socket\") pod \"35cccf52-b489-42d0-ae01-f3071bdc00ed\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") "
Oct 31 01:52:42.746923 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.746905    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35cccf52-b489-42d0-ae01-f3071bdc00ed-kube-api-access-27s4l\" (UniqueName: \"kubernetes.io/projected/35cccf52-b489-42d0-ae01-f3071bdc00ed-kube-api-access-27s4l\") pod \"35cccf52-b489-42d0-ae01-f3071bdc00ed\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") "
Oct 31 01:52:42.747096 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.747079    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-workload-certs\") pod \"35cccf52-b489-42d0-ae01-f3071bdc00ed\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") "
Oct 31 01:52:42.747216 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.747202    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-credential-socket\") pod \"35cccf52-b489-42d0-ae01-f3071bdc00ed\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") "
Oct 31 01:52:42.747397 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.747384    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-podinfo\") pod \"35cccf52-b489-42d0-ae01-f3071bdc00ed\" (UID: \"35cccf52-b489-42d0-ae01-f3071bdc00ed\") "
Oct 31 01:52:42.748255 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.748230    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cccf52-b489-42d0-ae01-f3071bdc00ed-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:42.748386 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.748371    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:42.749269 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.746861    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-workload-socket" pod "35cccf52-b489-42d0-ae01-f3071bdc00ed" (UID: "35cccf52-b489-42d0-ae01-f3071bdc00ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:42.749269 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.749191    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-credential-socket" pod "35cccf52-b489-42d0-ae01-f3071bdc00ed" (UID: "35cccf52-b489-42d0-ae01-f3071bdc00ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:42.749269 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.749246    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-workload-certs" pod "35cccf52-b489-42d0-ae01-f3071bdc00ed" (UID: "35cccf52-b489-42d0-ae01-f3071bdc00ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:42.752224 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.752199    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-podinfo" pod "35cccf52-b489-42d0-ae01-f3071bdc00ed" (UID: "35cccf52-b489-42d0-ae01-f3071bdc00ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:52:42.753419 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.753402    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35cccf52-b489-42d0-ae01-f3071bdc00ed-kube-api-access-27s4l" pod "35cccf52-b489-42d0-ae01-f3071bdc00ed" (UID: "35cccf52-b489-42d0-ae01-f3071bdc00ed"). InnerVolumeSpecName "kube-api-access-27s4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:42.755478 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.755428    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-token" pod "35cccf52-b489-42d0-ae01-f3071bdc00ed" (UID: "35cccf52-b489-42d0-ae01-f3071bdc00ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:42.757194 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.757166    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-envoy" pod "35cccf52-b489-42d0-ae01-f3071bdc00ed" (UID: "35cccf52-b489-42d0-ae01-f3071bdc00ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:42.850089 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.850053    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:42.850089 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.850088    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:42.850089 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.850097    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:42.850329 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.850107    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27s4l\" (UniqueName: \"kubernetes.io/projected/35cccf52-b489-42d0-ae01-f3071bdc00ed-kube-api-access-27s4l\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:42.850329 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.850117    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:42.850329 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.850125    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cccf52-b489-42d0-ae01-f3071bdc00ed-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:42.850329 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:42.850134    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cccf52-b489-42d0-ae01-f3071bdc00ed-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:43.035681 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.035647    5650 generic.go:356] "Generic (PLEG): container finished" podID="35cccf52-b489-42d0-ae01-f3071bdc00ed" containerID="75e24e03ff8c864169d6bebdd6ef912ef5e79bc626feea5471a72e6b2c1814cd" exitCode=0
Oct 31 01:52:43.035883 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.035692    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"
Oct 31 01:52:43.035883 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.035733    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp" event={"ID":"35cccf52-b489-42d0-ae01-f3071bdc00ed","Type":"ContainerDied","Data":"75e24e03ff8c864169d6bebdd6ef912ef5e79bc626feea5471a72e6b2c1814cd"}
Oct 31 01:52:43.035883 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.035792    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp" event={"ID":"35cccf52-b489-42d0-ae01-f3071bdc00ed","Type":"ContainerDied","Data":"8fd24fcb6ca495f1ca61e11d6ec89d9bf72344d26636527337afc2409bdde934"}
Oct 31 01:52:43.035883 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.035820    5650 scope.go:122] "RemoveContainer" containerID="75e24e03ff8c864169d6bebdd6ef912ef5e79bc626feea5471a72e6b2c1814cd"
Oct 31 01:52:43.050537 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.050508    5650 scope.go:122] "RemoveContainer" containerID="94491bea7057333a5d6b0d8796b2b74b2891ffaf050003e599b88fb4efb80fe7"
Oct 31 01:52:43.067201 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.067168    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"]
Oct 31 01:52:43.069101 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.069073    5650 scope.go:122] "RemoveContainer" containerID="75e24e03ff8c864169d6bebdd6ef912ef5e79bc626feea5471a72e6b2c1814cd"
Oct 31 01:52:43.069480 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:52:43.069457    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75e24e03ff8c864169d6bebdd6ef912ef5e79bc626feea5471a72e6b2c1814cd\": container with ID starting with 75e24e03ff8c864169d6bebdd6ef912ef5e79bc626feea5471a72e6b2c1814cd not found: ID does not exist" containerID="75e24e03ff8c864169d6bebdd6ef912ef5e79bc626feea5471a72e6b2c1814cd"
Oct 31 01:52:43.069575 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.069488    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75e24e03ff8c864169d6bebdd6ef912ef5e79bc626feea5471a72e6b2c1814cd"} err="failed to get container status \"75e24e03ff8c864169d6bebdd6ef912ef5e79bc626feea5471a72e6b2c1814cd\": rpc error: code = NotFound desc = could not find container \"75e24e03ff8c864169d6bebdd6ef912ef5e79bc626feea5471a72e6b2c1814cd\": container with ID starting with 75e24e03ff8c864169d6bebdd6ef912ef5e79bc626feea5471a72e6b2c1814cd not found: ID does not exist"
Oct 31 01:52:43.069575 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.069505    5650 scope.go:122] "RemoveContainer" containerID="94491bea7057333a5d6b0d8796b2b74b2891ffaf050003e599b88fb4efb80fe7"
Oct 31 01:52:43.069872 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:52:43.069855    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94491bea7057333a5d6b0d8796b2b74b2891ffaf050003e599b88fb4efb80fe7\": container with ID starting with 94491bea7057333a5d6b0d8796b2b74b2891ffaf050003e599b88fb4efb80fe7 not found: ID does not exist" containerID="94491bea7057333a5d6b0d8796b2b74b2891ffaf050003e599b88fb4efb80fe7"
Oct 31 01:52:43.069930 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.069876    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94491bea7057333a5d6b0d8796b2b74b2891ffaf050003e599b88fb4efb80fe7"} err="failed to get container status \"94491bea7057333a5d6b0d8796b2b74b2891ffaf050003e599b88fb4efb80fe7\": rpc error: code = NotFound desc = could not find container \"94491bea7057333a5d6b0d8796b2b74b2891ffaf050003e599b88fb4efb80fe7\": container with ID starting with 94491bea7057333a5d6b0d8796b2b74b2891ffaf050003e599b88fb4efb80fe7 not found: ID does not exist"
Oct 31 01:52:43.071349 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.071328    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-03004a-openshift-default-7f95b97cd7-rcppp"]
Oct 31 01:52:43.420531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.419823    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp" podUID="f7cdd7c3-0ba1-4558-8043-1be465a26b46" containerName="payload-pre-processing" containerID="cri-o://48b04a17852251e51d0298f8a09e4a4ca39c981d925561fea5cc7b385fd6fe57" gracePeriod=25
Oct 31 01:52:43.421118 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.420649    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t" podUID="1a64682c-f620-4031-8efd-d576b36a9161" containerName="payload-processing" containerID="cri-o://3706c689facde2b43b828d78bafd6824b466cefeeffe2bc85339c565daac52cb" gracePeriod=25
Oct 31 01:52:43.455094 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.455061    5650 patch_prober.go:29] interesting pod/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 31 01:52:43.455280 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.455119    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p" podUID="cb9db005-6073-4c28-bec1-36495261155c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 31 01:52:43.863626 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.863328    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp"
Oct 31 01:52:43.937926 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.937886    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t"
Oct 31 01:52:43.985190 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.985038    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7cdd7c3-0ba1-4558-8043-1be465a26b46-kube-api-access-4jrsp\" (UniqueName: \"kubernetes.io/projected/f7cdd7c3-0ba1-4558-8043-1be465a26b46-kube-api-access-4jrsp\") pod \"f7cdd7c3-0ba1-4558-8043-1be465a26b46\" (UID: \"f7cdd7c3-0ba1-4558-8043-1be465a26b46\") "
Oct 31 01:52:43.985190 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.985104    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a64682c-f620-4031-8efd-d576b36a9161-kube-api-access-b7ds6\" (UniqueName: \"kubernetes.io/projected/1a64682c-f620-4031-8efd-d576b36a9161-kube-api-access-b7ds6\") pod \"1a64682c-f620-4031-8efd-d576b36a9161\" (UID: \"1a64682c-f620-4031-8efd-d576b36a9161\") "
Oct 31 01:52:43.985435 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.985225    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a64682c-f620-4031-8efd-d576b36a9161-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a64682c-f620-4031-8efd-d576b36a9161-plugins-config-volume\") pod \"1a64682c-f620-4031-8efd-d576b36a9161\" (UID: \"1a64682c-f620-4031-8efd-d576b36a9161\") "
Oct 31 01:52:43.985435 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.985359    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7cdd7c3-0ba1-4558-8043-1be465a26b46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7cdd7c3-0ba1-4558-8043-1be465a26b46-plugins-config-volume\") pod \"f7cdd7c3-0ba1-4558-8043-1be465a26b46\" (UID: \"f7cdd7c3-0ba1-4558-8043-1be465a26b46\") "
Oct 31 01:52:43.986513 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.986482    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7cdd7c3-0ba1-4558-8043-1be465a26b46-plugins-config-volume" pod "f7cdd7c3-0ba1-4558-8043-1be465a26b46" (UID: "f7cdd7c3-0ba1-4558-8043-1be465a26b46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:43.986857 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.986830    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a64682c-f620-4031-8efd-d576b36a9161-plugins-config-volume" pod "1a64682c-f620-4031-8efd-d576b36a9161" (UID: "1a64682c-f620-4031-8efd-d576b36a9161"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:43.988409 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.988381    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a64682c-f620-4031-8efd-d576b36a9161-kube-api-access-b7ds6" pod "1a64682c-f620-4031-8efd-d576b36a9161" (UID: "1a64682c-f620-4031-8efd-d576b36a9161"). InnerVolumeSpecName "kube-api-access-b7ds6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:43.989752 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:43.989728    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7cdd7c3-0ba1-4558-8043-1be465a26b46-kube-api-access-4jrsp" pod "f7cdd7c3-0ba1-4558-8043-1be465a26b46" (UID: "f7cdd7c3-0ba1-4558-8043-1be465a26b46"). InnerVolumeSpecName "kube-api-access-4jrsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:44.050027 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.049987    5650 generic.go:356] "Generic (PLEG): container finished" podID="f7cdd7c3-0ba1-4558-8043-1be465a26b46" containerID="48b04a17852251e51d0298f8a09e4a4ca39c981d925561fea5cc7b385fd6fe57" exitCode=0
Oct 31 01:52:44.050200 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.050094    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp" event={"ID":"f7cdd7c3-0ba1-4558-8043-1be465a26b46","Type":"ContainerDied","Data":"48b04a17852251e51d0298f8a09e4a4ca39c981d925561fea5cc7b385fd6fe57"}
Oct 31 01:52:44.050200 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.050174    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp" event={"ID":"f7cdd7c3-0ba1-4558-8043-1be465a26b46","Type":"ContainerDied","Data":"4130bdf0df30889ed723b65ee0338d95351cd24273382641e9d9a6bf1091c231"}
Oct 31 01:52:44.050329 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.050202    5650 scope.go:122] "RemoveContainer" containerID="48b04a17852251e51d0298f8a09e4a4ca39c981d925561fea5cc7b385fd6fe57"
Oct 31 01:52:44.050375 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.050361    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp"
Oct 31 01:52:44.055722 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.055690    5650 generic.go:356] "Generic (PLEG): container finished" podID="1a64682c-f620-4031-8efd-d576b36a9161" containerID="3706c689facde2b43b828d78bafd6824b466cefeeffe2bc85339c565daac52cb" exitCode=0
Oct 31 01:52:44.056167 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.055970    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t"
Oct 31 01:52:44.059645 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.056008    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t" event={"ID":"1a64682c-f620-4031-8efd-d576b36a9161","Type":"ContainerDied","Data":"3706c689facde2b43b828d78bafd6824b466cefeeffe2bc85339c565daac52cb"}
Oct 31 01:52:44.059645 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.059567    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t" event={"ID":"1a64682c-f620-4031-8efd-d576b36a9161","Type":"ContainerDied","Data":"37810afd152d95b154087daa18673a73a3947173ef4227a4a5119f9e8f3babb1"}
Oct 31 01:52:44.069338 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.069290    5650 scope.go:122] "RemoveContainer" containerID="48b04a17852251e51d0298f8a09e4a4ca39c981d925561fea5cc7b385fd6fe57"
Oct 31 01:52:44.069751 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:52:44.069723    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48b04a17852251e51d0298f8a09e4a4ca39c981d925561fea5cc7b385fd6fe57\": container with ID starting with 48b04a17852251e51d0298f8a09e4a4ca39c981d925561fea5cc7b385fd6fe57 not found: ID does not exist" containerID="48b04a17852251e51d0298f8a09e4a4ca39c981d925561fea5cc7b385fd6fe57"
Oct 31 01:52:44.069900 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.069762    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48b04a17852251e51d0298f8a09e4a4ca39c981d925561fea5cc7b385fd6fe57"} err="failed to get container status \"48b04a17852251e51d0298f8a09e4a4ca39c981d925561fea5cc7b385fd6fe57\": rpc error: code = NotFound desc = could not find container \"48b04a17852251e51d0298f8a09e4a4ca39c981d925561fea5cc7b385fd6fe57\": container with ID starting with 48b04a17852251e51d0298f8a09e4a4ca39c981d925561fea5cc7b385fd6fe57 not found: ID does not exist"
Oct 31 01:52:44.069900 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.069787    5650 scope.go:122] "RemoveContainer" containerID="3706c689facde2b43b828d78bafd6824b466cefeeffe2bc85339c565daac52cb"
Oct 31 01:52:44.085858 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.085820    5650 scope.go:122] "RemoveContainer" containerID="3706c689facde2b43b828d78bafd6824b466cefeeffe2bc85339c565daac52cb"
Oct 31 01:52:44.086655 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:52:44.086621    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3706c689facde2b43b828d78bafd6824b466cefeeffe2bc85339c565daac52cb\": container with ID starting with 3706c689facde2b43b828d78bafd6824b466cefeeffe2bc85339c565daac52cb not found: ID does not exist" containerID="3706c689facde2b43b828d78bafd6824b466cefeeffe2bc85339c565daac52cb"
Oct 31 01:52:44.086761 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.086668    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3706c689facde2b43b828d78bafd6824b466cefeeffe2bc85339c565daac52cb"} err="failed to get container status \"3706c689facde2b43b828d78bafd6824b466cefeeffe2bc85339c565daac52cb\": rpc error: code = NotFound desc = could not find container \"3706c689facde2b43b828d78bafd6824b466cefeeffe2bc85339c565daac52cb\": container with ID starting with 3706c689facde2b43b828d78bafd6824b466cefeeffe2bc85339c565daac52cb not found: ID does not exist"
Oct 31 01:52:44.088391 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.088365    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp"]
Oct 31 01:52:44.088832 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.088805    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a64682c-f620-4031-8efd-d576b36a9161-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:44.088906 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.088849    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7cdd7c3-0ba1-4558-8043-1be465a26b46-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:44.088906 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.088866    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jrsp\" (UniqueName: \"kubernetes.io/projected/f7cdd7c3-0ba1-4558-8043-1be465a26b46-kube-api-access-4jrsp\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:44.088906 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.088880    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7ds6\" (UniqueName: \"kubernetes.io/projected/1a64682c-f620-4031-8efd-d576b36a9161-kube-api-access-b7ds6\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:44.092196 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.092175    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b8d890-7588c66bcf-pvfnp"]
Oct 31 01:52:44.136603 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.136563    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35cccf52-b489-42d0-ae01-f3071bdc00ed" path="/var/lib/kubelet/pods/35cccf52-b489-42d0-ae01-f3071bdc00ed/volumes"
Oct 31 01:52:44.137299 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.137278    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7cdd7c3-0ba1-4558-8043-1be465a26b46" path="/var/lib/kubelet/pods/f7cdd7c3-0ba1-4558-8043-1be465a26b46/volumes"
Oct 31 01:52:44.137751 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.137732    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t"]
Oct 31 01:52:44.141139 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.141108    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b8d890-76f99c4c5c-r8p4t"]
Oct 31 01:52:44.719918 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.719877    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:44.720348 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.720321    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:44.721970 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:44.721930    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:46.124680 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:46.124645    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a64682c-f620-4031-8efd-d576b36a9161" path="/var/lib/kubelet/pods/1a64682c-f620-4031-8efd-d576b36a9161/volumes"
Oct 31 01:52:47.967164 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:47.967127    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"]
Oct 31 01:52:47.967692 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:47.967357    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p" podUID="cb9db005-6073-4c28-bec1-36495261155c" containerName="istio-proxy" containerID="cri-o://745fced1f9af15956a1344fbb6497548dc53fc84d6218f4936d361d917615e5b" gracePeriod=30
Oct 31 01:52:48.879222 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:48.879184    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7"]
Oct 31 01:52:48.898496 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:48.894840    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6"]
Oct 31 01:52:53.191987 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.191947    5650 generic.go:356] "Generic (PLEG): container finished" podID="cb9db005-6073-4c28-bec1-36495261155c" containerID="745fced1f9af15956a1344fbb6497548dc53fc84d6218f4936d361d917615e5b" exitCode=0
Oct 31 01:52:53.192602 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.192010    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p" event={"ID":"cb9db005-6073-4c28-bec1-36495261155c","Type":"ContainerDied","Data":"745fced1f9af15956a1344fbb6497548dc53fc84d6218f4936d361d917615e5b"}
Oct 31 01:52:53.263269 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.263243    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:53.291153 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.291122    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb9db005-6073-4c28-bec1-36495261155c-kube-api-access-x86tp\" (UniqueName: \"kubernetes.io/projected/cb9db005-6073-4c28-bec1-36495261155c-kube-api-access-x86tp\") pod \"cb9db005-6073-4c28-bec1-36495261155c\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") "
Oct 31 01:52:53.291327 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.291262    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-workload-socket\") pod \"cb9db005-6073-4c28-bec1-36495261155c\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") "
Oct 31 01:52:53.291327 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.291300    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-istio-data\") pod \"cb9db005-6073-4c28-bec1-36495261155c\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") "
Oct 31 01:52:53.291482 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.291350    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb9db005-6073-4c28-bec1-36495261155c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb9db005-6073-4c28-bec1-36495261155c-istiod-ca-cert\") pod \"cb9db005-6073-4c28-bec1-36495261155c\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") "
Oct 31 01:52:53.291482 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.291411    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-credential-socket\") pod \"cb9db005-6073-4c28-bec1-36495261155c\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") "
Oct 31 01:52:53.291482 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.291461    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb9db005-6073-4c28-bec1-36495261155c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb9db005-6073-4c28-bec1-36495261155c-istio-podinfo\") pod \"cb9db005-6073-4c28-bec1-36495261155c\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") "
Oct 31 01:52:53.291646 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.291518    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb9db005-6073-4c28-bec1-36495261155c-istio-token\" (UniqueName: \"kubernetes.io/projected/cb9db005-6073-4c28-bec1-36495261155c-istio-token\") pod \"cb9db005-6073-4c28-bec1-36495261155c\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") "
Oct 31 01:52:53.291646 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.291631    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-istio-envoy\") pod \"cb9db005-6073-4c28-bec1-36495261155c\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") "
Oct 31 01:52:53.291747 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.291723    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-workload-certs\") pod \"cb9db005-6073-4c28-bec1-36495261155c\" (UID: \"cb9db005-6073-4c28-bec1-36495261155c\") "
Oct 31 01:52:53.292860 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.292024    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb9db005-6073-4c28-bec1-36495261155c-istiod-ca-cert" pod "cb9db005-6073-4c28-bec1-36495261155c" (UID: "cb9db005-6073-4c28-bec1-36495261155c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:53.292860 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.292257    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-workload-certs" pod "cb9db005-6073-4c28-bec1-36495261155c" (UID: "cb9db005-6073-4c28-bec1-36495261155c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:53.292860 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.292433    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-credential-socket" pod "cb9db005-6073-4c28-bec1-36495261155c" (UID: "cb9db005-6073-4c28-bec1-36495261155c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:53.292860 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.292656    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-workload-socket" pod "cb9db005-6073-4c28-bec1-36495261155c" (UID: "cb9db005-6073-4c28-bec1-36495261155c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:53.292860 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.292726    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb9db005-6073-4c28-bec1-36495261155c-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:53.292860 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.292750    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:53.292860 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.292766    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:53.292860 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.292834    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-istio-data" pod "cb9db005-6073-4c28-bec1-36495261155c" (UID: "cb9db005-6073-4c28-bec1-36495261155c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:53.307949 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.307909    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb9db005-6073-4c28-bec1-36495261155c-istio-podinfo" pod "cb9db005-6073-4c28-bec1-36495261155c" (UID: "cb9db005-6073-4c28-bec1-36495261155c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:52:53.308105 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.307962    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb9db005-6073-4c28-bec1-36495261155c-kube-api-access-x86tp" pod "cb9db005-6073-4c28-bec1-36495261155c" (UID: "cb9db005-6073-4c28-bec1-36495261155c"). InnerVolumeSpecName "kube-api-access-x86tp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:53.308105 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.308000    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb9db005-6073-4c28-bec1-36495261155c-istio-token" pod "cb9db005-6073-4c28-bec1-36495261155c" (UID: "cb9db005-6073-4c28-bec1-36495261155c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:53.313240 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.313194    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-istio-envoy" pod "cb9db005-6073-4c28-bec1-36495261155c" (UID: "cb9db005-6073-4c28-bec1-36495261155c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:53.394586 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.394549    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:53.394586 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.394584    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x86tp\" (UniqueName: \"kubernetes.io/projected/cb9db005-6073-4c28-bec1-36495261155c-kube-api-access-x86tp\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:53.394817 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.394598    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:53.394817 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.394611    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb9db005-6073-4c28-bec1-36495261155c-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:53.394817 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.394624    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb9db005-6073-4c28-bec1-36495261155c-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:53.394817 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.394636    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb9db005-6073-4c28-bec1-36495261155c-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:53.880186 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.880124    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7" podUID="fe46844f-a0d6-4210-8516-f4facfbe165f" containerName="payload-processing" containerID="cri-o://483d7abc0bb3d588e2fa0530e76682ab6027a3a530ca8ddcb07bf175ef9c7d09" gracePeriod=25
Oct 31 01:52:53.895284 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:53.895241    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6" podUID="840dad17-e4e9-4a19-83d4-af8ce9ea1aad" containerName="payload-pre-processing" containerID="cri-o://3fad9e812ad79c439393ed5082cb63e12852c9b801c8d09ba3b1501851340652" gracePeriod=25
Oct 31 01:52:54.105501 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.096167    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd"]
Oct 31 01:52:54.114372 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.113783    5650 cpu_manager.go:414] "containerMap: removing container" podUID="35cccf52-b489-42d0-ae01-f3071bdc00ed" containerName="istio-proxy"
Oct 31 01:52:54.114372 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.114138    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35cccf52-b489-42d0-ae01-f3071bdc00ed" containerName="istio-proxy"
Oct 31 01:52:54.114372 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.114230    5650 cpu_manager.go:414] "containerMap: removing container" podUID="1a64682c-f620-4031-8efd-d576b36a9161" containerName="payload-processing"
Oct 31 01:52:54.114372 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.114251    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a64682c-f620-4031-8efd-d576b36a9161" containerName="payload-processing"
Oct 31 01:52:54.114372 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.114288    5650 cpu_manager.go:414] "containerMap: removing container" podUID="cb9db005-6073-4c28-bec1-36495261155c" containerName="istio-proxy"
Oct 31 01:52:54.114372 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.114296    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb9db005-6073-4c28-bec1-36495261155c" containerName="istio-proxy"
Oct 31 01:52:54.114372 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.114342    5650 cpu_manager.go:414] "containerMap: removing container" podUID="f7cdd7c3-0ba1-4558-8043-1be465a26b46" containerName="payload-pre-processing"
Oct 31 01:52:54.114372 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.114352    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7cdd7c3-0ba1-4558-8043-1be465a26b46" containerName="payload-pre-processing"
Oct 31 01:52:54.114919 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.114899    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="35cccf52-b489-42d0-ae01-f3071bdc00ed" containerName="istio-proxy"
Oct 31 01:52:54.115049 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.115024    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a64682c-f620-4031-8efd-d576b36a9161" containerName="payload-processing"
Oct 31 01:52:54.115132 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.115122    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb9db005-6073-4c28-bec1-36495261155c" containerName="istio-proxy"
Oct 31 01:52:54.115201 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.115194    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7cdd7c3-0ba1-4558-8043-1be465a26b46" containerName="payload-pre-processing"
Oct 31 01:52:54.124425 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.123664    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd"]
Oct 31 01:52:54.124425 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.123853    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd"
Oct 31 01:52:54.133960 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.133884    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-925bac-dockercfg-xg4kz\""
Oct 31 01:52:54.138595 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.138567    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5"]
Oct 31 01:52:54.139588 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.139561    5650 cpu_manager.go:414] "containerMap: removing container" podUID="35cccf52-b489-42d0-ae01-f3071bdc00ed" containerName="istio-proxy"
Oct 31 01:52:54.139588 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.139588    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35cccf52-b489-42d0-ae01-f3071bdc00ed" containerName="istio-proxy"
Oct 31 01:52:54.139817 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.139804    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="35cccf52-b489-42d0-ae01-f3071bdc00ed" containerName="istio-proxy"
Oct 31 01:52:54.147310 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.145876    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5"
Oct 31 01:52:54.157955 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.157921    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5"]
Oct 31 01:52:54.220520 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.220486    5650 generic.go:356] "Generic (PLEG): container finished" podID="fe46844f-a0d6-4210-8516-f4facfbe165f" containerID="483d7abc0bb3d588e2fa0530e76682ab6027a3a530ca8ddcb07bf175ef9c7d09" exitCode=0
Oct 31 01:52:54.221044 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.220613    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7" event={"ID":"fe46844f-a0d6-4210-8516-f4facfbe165f","Type":"ContainerDied","Data":"483d7abc0bb3d588e2fa0530e76682ab6027a3a530ca8ddcb07bf175ef9c7d09"}
Oct 31 01:52:54.221044 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.220665    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7" event={"ID":"fe46844f-a0d6-4210-8516-f4facfbe165f","Type":"ContainerDied","Data":"f18b04f9128ec98e1a764b8f2303fef06a69532a3f1cb540540029f48896f67c"}
Oct 31 01:52:54.221044 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.220689    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f18b04f9128ec98e1a764b8f2303fef06a69532a3f1cb540540029f48896f67c"
Oct 31 01:52:54.222344 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.222313    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44xjv\" (UniqueName: \"kubernetes.io/projected/be03a565-244e-4052-a13c-97bf29318774-kube-api-access-44xjv\") pod \"payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5\" (UID: \"be03a565-244e-4052-a13c-97bf29318774\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5"
Oct 31 01:52:54.222478 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.222389    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be03a565-244e-4052-a13c-97bf29318774-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5\" (UID: \"be03a565-244e-4052-a13c-97bf29318774\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5"
Oct 31 01:52:54.222558 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.222538    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd7bede-09f7-45a6-9615-d98483151c9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd\" (UID: \"7bd7bede-09f7-45a6-9615-d98483151c9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd"
Oct 31 01:52:54.222618 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.222601    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvl47\" (UniqueName: \"kubernetes.io/projected/7bd7bede-09f7-45a6-9615-d98483151c9f-kube-api-access-qvl47\") pod \"payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd\" (UID: \"7bd7bede-09f7-45a6-9615-d98483151c9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd"
Oct 31 01:52:54.227221 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.227192    5650 generic.go:356] "Generic (PLEG): container finished" podID="840dad17-e4e9-4a19-83d4-af8ce9ea1aad" containerID="3fad9e812ad79c439393ed5082cb63e12852c9b801c8d09ba3b1501851340652" exitCode=0
Oct 31 01:52:54.227345 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.227305    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6" event={"ID":"840dad17-e4e9-4a19-83d4-af8ce9ea1aad","Type":"ContainerDied","Data":"3fad9e812ad79c439393ed5082cb63e12852c9b801c8d09ba3b1501851340652"}
Oct 31 01:52:54.229063 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.229038    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"
Oct 31 01:52:54.229211 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.229033    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p" event={"ID":"cb9db005-6073-4c28-bec1-36495261155c","Type":"ContainerDied","Data":"59063a370c3a0211500958a1cc4808b61cd87cd1df0ce4254262745b62cf91c6"}
Oct 31 01:52:54.229273 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.229211    5650 scope.go:122] "RemoveContainer" containerID="745fced1f9af15956a1344fbb6497548dc53fc84d6218f4936d361d917615e5b"
Oct 31 01:52:54.229624 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.229604    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7"
Oct 31 01:52:54.286860 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.286820    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"]
Oct 31 01:52:54.292150 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.292117    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-msc7p"]
Oct 31 01:52:54.311008 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.310284    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6"
Oct 31 01:52:54.326398 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.326363    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe46844f-a0d6-4210-8516-f4facfbe165f-kube-api-access-6ls6t\" (UniqueName: \"kubernetes.io/projected/fe46844f-a0d6-4210-8516-f4facfbe165f-kube-api-access-6ls6t\") pod \"fe46844f-a0d6-4210-8516-f4facfbe165f\" (UID: \"fe46844f-a0d6-4210-8516-f4facfbe165f\") "
Oct 31 01:52:54.326612 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.326544    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe46844f-a0d6-4210-8516-f4facfbe165f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe46844f-a0d6-4210-8516-f4facfbe165f-plugins-config-volume\") pod \"fe46844f-a0d6-4210-8516-f4facfbe165f\" (UID: \"fe46844f-a0d6-4210-8516-f4facfbe165f\") "
Oct 31 01:52:54.326927 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.326900    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd7bede-09f7-45a6-9615-d98483151c9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd\" (UID: \"7bd7bede-09f7-45a6-9615-d98483151c9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd"
Oct 31 01:52:54.327148 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.327131    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvl47\" (UniqueName: \"kubernetes.io/projected/7bd7bede-09f7-45a6-9615-d98483151c9f-kube-api-access-qvl47\") pod \"payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd\" (UID: \"7bd7bede-09f7-45a6-9615-d98483151c9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd"
Oct 31 01:52:54.327503 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.327484    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44xjv\" (UniqueName: \"kubernetes.io/projected/be03a565-244e-4052-a13c-97bf29318774-kube-api-access-44xjv\") pod \"payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5\" (UID: \"be03a565-244e-4052-a13c-97bf29318774\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5"
Oct 31 01:52:54.327682 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.327667    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be03a565-244e-4052-a13c-97bf29318774-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5\" (UID: \"be03a565-244e-4052-a13c-97bf29318774\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5"
Oct 31 01:52:54.328004 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.327511    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe46844f-a0d6-4210-8516-f4facfbe165f-plugins-config-volume" pod "fe46844f-a0d6-4210-8516-f4facfbe165f" (UID: "fe46844f-a0d6-4210-8516-f4facfbe165f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:54.328093 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.328053    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd7bede-09f7-45a6-9615-d98483151c9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd\" (UID: \"7bd7bede-09f7-45a6-9615-d98483151c9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd"
Oct 31 01:52:54.328932 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.328911    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be03a565-244e-4052-a13c-97bf29318774-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5\" (UID: \"be03a565-244e-4052-a13c-97bf29318774\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5"
Oct 31 01:52:54.332256 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.332234    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe46844f-a0d6-4210-8516-f4facfbe165f-kube-api-access-6ls6t" pod "fe46844f-a0d6-4210-8516-f4facfbe165f" (UID: "fe46844f-a0d6-4210-8516-f4facfbe165f"). InnerVolumeSpecName "kube-api-access-6ls6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:54.369498 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.369438    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvl47\" (UniqueName: \"kubernetes.io/projected/7bd7bede-09f7-45a6-9615-d98483151c9f-kube-api-access-qvl47\") pod \"payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd\" (UID: \"7bd7bede-09f7-45a6-9615-d98483151c9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd"
Oct 31 01:52:54.371177 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.371143    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44xjv\" (UniqueName: \"kubernetes.io/projected/be03a565-244e-4052-a13c-97bf29318774-kube-api-access-44xjv\") pod \"payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5\" (UID: \"be03a565-244e-4052-a13c-97bf29318774\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5"
Oct 31 01:52:54.428961 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.428878    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/840dad17-e4e9-4a19-83d4-af8ce9ea1aad-kube-api-access-57c2b\" (UniqueName: \"kubernetes.io/projected/840dad17-e4e9-4a19-83d4-af8ce9ea1aad-kube-api-access-57c2b\") pod \"840dad17-e4e9-4a19-83d4-af8ce9ea1aad\" (UID: \"840dad17-e4e9-4a19-83d4-af8ce9ea1aad\") "
Oct 31 01:52:54.429100 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.428989    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/840dad17-e4e9-4a19-83d4-af8ce9ea1aad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/840dad17-e4e9-4a19-83d4-af8ce9ea1aad-plugins-config-volume\") pod \"840dad17-e4e9-4a19-83d4-af8ce9ea1aad\" (UID: \"840dad17-e4e9-4a19-83d4-af8ce9ea1aad\") "
Oct 31 01:52:54.429720 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.429691    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ls6t\" (UniqueName: \"kubernetes.io/projected/fe46844f-a0d6-4210-8516-f4facfbe165f-kube-api-access-6ls6t\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:54.429720 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.429722    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe46844f-a0d6-4210-8516-f4facfbe165f-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:54.429923 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.429744    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/840dad17-e4e9-4a19-83d4-af8ce9ea1aad-plugins-config-volume" pod "840dad17-e4e9-4a19-83d4-af8ce9ea1aad" (UID: "840dad17-e4e9-4a19-83d4-af8ce9ea1aad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:54.432490 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.431777    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/840dad17-e4e9-4a19-83d4-af8ce9ea1aad-kube-api-access-57c2b" pod "840dad17-e4e9-4a19-83d4-af8ce9ea1aad" (UID: "840dad17-e4e9-4a19-83d4-af8ce9ea1aad"). InnerVolumeSpecName "kube-api-access-57c2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:54.445754 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.445721    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd"
Oct 31 01:52:54.476531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.468770    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5"
Oct 31 01:52:54.532130 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.532058    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57c2b\" (UniqueName: \"kubernetes.io/projected/840dad17-e4e9-4a19-83d4-af8ce9ea1aad-kube-api-access-57c2b\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:54.532130 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.532100    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/840dad17-e4e9-4a19-83d4-af8ce9ea1aad-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:52:54.788305 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.787875    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5"]
Oct 31 01:52:54.793097 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:52:54.793006    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd7bede_09f7_45a6_9615_d98483151c9f.slice/crio-9765d67b0731c142a749ded551c168f66748302f2890226c23818bd0a6c54ce8 WatchSource:0}: Error finding container 9765d67b0731c142a749ded551c168f66748302f2890226c23818bd0a6c54ce8: Status 404 returned error can't find the container with id 9765d67b0731c142a749ded551c168f66748302f2890226c23818bd0a6c54ce8
Oct 31 01:52:54.793433 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:54.793409    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd"]
Oct 31 01:52:55.244398 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:55.244365    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6"
Oct 31 01:52:55.244886 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:55.244364    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6" event={"ID":"840dad17-e4e9-4a19-83d4-af8ce9ea1aad","Type":"ContainerDied","Data":"74978ad36a4c5f811b2b1871e584e2ff2ac4dbacb8061f862bfa9a4ff619c78a"}
Oct 31 01:52:55.244886 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:55.244536    5650 scope.go:122] "RemoveContainer" containerID="3fad9e812ad79c439393ed5082cb63e12852c9b801c8d09ba3b1501851340652"
Oct 31 01:52:55.247180 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:55.247144    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5" event={"ID":"be03a565-244e-4052-a13c-97bf29318774","Type":"ContainerStarted","Data":"989c07952ffe3bdc31a0ac1e409337cb942db5b6f9793c873e2f93ba60de7fe4"}
Oct 31 01:52:55.247318 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:55.247214    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5" event={"ID":"be03a565-244e-4052-a13c-97bf29318774","Type":"ContainerStarted","Data":"f0376b21ec25167f3ca71509ef4e158a613b9691a72511489c42e0a6a951cbe0"}
Oct 31 01:52:55.248216 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:55.248190    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5"
Oct 31 01:52:55.252982 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:55.252800    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd" event={"ID":"7bd7bede-09f7-45a6-9615-d98483151c9f","Type":"ContainerStarted","Data":"62c5e76358dfadfdf118f6447974d78b60f83c4f26168c055b54fbc691b1a9dc"}
Oct 31 01:52:55.252982 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:55.252859    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd" event={"ID":"7bd7bede-09f7-45a6-9615-d98483151c9f","Type":"ContainerStarted","Data":"9765d67b0731c142a749ded551c168f66748302f2890226c23818bd0a6c54ce8"}
Oct 31 01:52:55.252982 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:55.252868    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7"
Oct 31 01:52:55.295816 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:55.295759    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5" podStartSLOduration=1.29573879 podStartE2EDuration="1.29573879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:55.274250731 +0000 UTC m=+1931.277965023" watchObservedRunningTime="2026-10-31 01:52:55.29573879 +0000 UTC m=+1931.299453083"
Oct 31 01:52:55.297073 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:55.297036    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd" podStartSLOduration=1.297024869 podStartE2EDuration="1.297024869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:55.294523207 +0000 UTC m=+1931.298237503" watchObservedRunningTime="2026-10-31 01:52:55.297024869 +0000 UTC m=+1931.300739161"
Oct 31 01:52:55.321127 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:55.321083    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6"]
Oct 31 01:52:55.326715 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:55.326682    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-5f4fc787c5-2t4h6"]
Oct 31 01:52:55.338393 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:55.338355    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7"]
Oct 31 01:52:55.358748 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:55.358706    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-925bac-67c8b68c9-j7xf7"]
Oct 31 01:52:56.137745 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:56.137707    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="840dad17-e4e9-4a19-83d4-af8ce9ea1aad" path="/var/lib/kubelet/pods/840dad17-e4e9-4a19-83d4-af8ce9ea1aad/volumes"
Oct 31 01:52:56.138420 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:56.138391    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb9db005-6073-4c28-bec1-36495261155c" path="/var/lib/kubelet/pods/cb9db005-6073-4c28-bec1-36495261155c/volumes"
Oct 31 01:52:56.138996 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:56.138972    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe46844f-a0d6-4210-8516-f4facfbe165f" path="/var/lib/kubelet/pods/fe46844f-a0d6-4210-8516-f4facfbe165f/volumes"
Oct 31 01:52:56.266612 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:56.266583    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd"
Oct 31 01:52:58.717469 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:52:58.704617    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:52:58.857471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:58.857233    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf"]
Oct 31 01:52:58.877629 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:52:58.876087    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft"]
Oct 31 01:53:03.857731 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:03.857680    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf" podUID="786fd80e-b891-4ccc-a776-069b6f6ba3f9" containerName="payload-processing" containerID="cri-o://aad3384e27760598dc81cc9262ec53bf8236bcd4a61d40f3cc16910cb7d5eabe" gracePeriod=25
Oct 31 01:53:03.876882 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:03.876803    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft" podUID="22a469a4-b8b4-4601-a954-a992c76be571" containerName="payload-pre-processing" containerID="cri-o://5a21a20c2f28ed5c46d99e3b0116a73d83902da91ae618de611cf55bbab1f83a" gracePeriod=25
Oct 31 01:53:04.243347 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.243316    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf"
Oct 31 01:53:04.263015 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.262983    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft"
Oct 31 01:53:04.331841 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.331805    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/786fd80e-b891-4ccc-a776-069b6f6ba3f9-kube-api-access-d559k\" (UniqueName: \"kubernetes.io/projected/786fd80e-b891-4ccc-a776-069b6f6ba3f9-kube-api-access-d559k\") pod \"786fd80e-b891-4ccc-a776-069b6f6ba3f9\" (UID: \"786fd80e-b891-4ccc-a776-069b6f6ba3f9\") "
Oct 31 01:53:04.332020 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.332002    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/786fd80e-b891-4ccc-a776-069b6f6ba3f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/786fd80e-b891-4ccc-a776-069b6f6ba3f9-plugins-config-volume\") pod \"786fd80e-b891-4ccc-a776-069b6f6ba3f9\" (UID: \"786fd80e-b891-4ccc-a776-069b6f6ba3f9\") "
Oct 31 01:53:04.332374 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.332352    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/786fd80e-b891-4ccc-a776-069b6f6ba3f9-plugins-config-volume" pod "786fd80e-b891-4ccc-a776-069b6f6ba3f9" (UID: "786fd80e-b891-4ccc-a776-069b6f6ba3f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:04.332683 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.332664    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/786fd80e-b891-4ccc-a776-069b6f6ba3f9-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:04.334160 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.334129    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/786fd80e-b891-4ccc-a776-069b6f6ba3f9-kube-api-access-d559k" pod "786fd80e-b891-4ccc-a776-069b6f6ba3f9" (UID: "786fd80e-b891-4ccc-a776-069b6f6ba3f9"). InnerVolumeSpecName "kube-api-access-d559k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:04.434520 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.433680    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22a469a4-b8b4-4601-a954-a992c76be571-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22a469a4-b8b4-4601-a954-a992c76be571-plugins-config-volume\") pod \"22a469a4-b8b4-4601-a954-a992c76be571\" (UID: \"22a469a4-b8b4-4601-a954-a992c76be571\") "
Oct 31 01:53:04.434520 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.433848    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a469a4-b8b4-4601-a954-a992c76be571-kube-api-access-ltj7s\" (UniqueName: \"kubernetes.io/projected/22a469a4-b8b4-4601-a954-a992c76be571-kube-api-access-ltj7s\") pod \"22a469a4-b8b4-4601-a954-a992c76be571\" (UID: \"22a469a4-b8b4-4601-a954-a992c76be571\") "
Oct 31 01:53:04.434520 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.434228    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d559k\" (UniqueName: \"kubernetes.io/projected/786fd80e-b891-4ccc-a776-069b6f6ba3f9-kube-api-access-d559k\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:04.439159 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.438040    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a469a4-b8b4-4601-a954-a992c76be571-plugins-config-volume" pod "22a469a4-b8b4-4601-a954-a992c76be571" (UID: "22a469a4-b8b4-4601-a954-a992c76be571"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:04.441978 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.441936    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a469a4-b8b4-4601-a954-a992c76be571-kube-api-access-ltj7s" pod "22a469a4-b8b4-4601-a954-a992c76be571" (UID: "22a469a4-b8b4-4601-a954-a992c76be571"). InnerVolumeSpecName "kube-api-access-ltj7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:04.443506 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.443371    5650 generic.go:356] "Generic (PLEG): container finished" podID="786fd80e-b891-4ccc-a776-069b6f6ba3f9" containerID="aad3384e27760598dc81cc9262ec53bf8236bcd4a61d40f3cc16910cb7d5eabe" exitCode=0
Oct 31 01:53:04.443740 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.443708    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf" event={"ID":"786fd80e-b891-4ccc-a776-069b6f6ba3f9","Type":"ContainerDied","Data":"aad3384e27760598dc81cc9262ec53bf8236bcd4a61d40f3cc16910cb7d5eabe"}
Oct 31 01:53:04.443861 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.443776    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf" event={"ID":"786fd80e-b891-4ccc-a776-069b6f6ba3f9","Type":"ContainerDied","Data":"0663cfcff8de6e27b734bed7d2e77dee1004e2e362f615cc1dac84b4d2aea821"}
Oct 31 01:53:04.443861 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.443810    5650 scope.go:122] "RemoveContainer" containerID="aad3384e27760598dc81cc9262ec53bf8236bcd4a61d40f3cc16910cb7d5eabe"
Oct 31 01:53:04.443988 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.443971    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf"
Oct 31 01:53:04.450040 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.450009    5650 generic.go:356] "Generic (PLEG): container finished" podID="22a469a4-b8b4-4601-a954-a992c76be571" containerID="5a21a20c2f28ed5c46d99e3b0116a73d83902da91ae618de611cf55bbab1f83a" exitCode=0
Oct 31 01:53:04.450254 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.450235    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft" event={"ID":"22a469a4-b8b4-4601-a954-a992c76be571","Type":"ContainerDied","Data":"5a21a20c2f28ed5c46d99e3b0116a73d83902da91ae618de611cf55bbab1f83a"}
Oct 31 01:53:04.450348 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.450285    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft" event={"ID":"22a469a4-b8b4-4601-a954-a992c76be571","Type":"ContainerDied","Data":"45985ffc0e82522aa529019c34ac3cfaed77b62cbab66af70f6baf1179f7dceb"}
Oct 31 01:53:04.450410 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.450354    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft"
Oct 31 01:53:04.469049 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.468768    5650 scope.go:122] "RemoveContainer" containerID="aad3384e27760598dc81cc9262ec53bf8236bcd4a61d40f3cc16910cb7d5eabe"
Oct 31 01:53:04.469574 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:53:04.469535    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aad3384e27760598dc81cc9262ec53bf8236bcd4a61d40f3cc16910cb7d5eabe\": container with ID starting with aad3384e27760598dc81cc9262ec53bf8236bcd4a61d40f3cc16910cb7d5eabe not found: ID does not exist" containerID="aad3384e27760598dc81cc9262ec53bf8236bcd4a61d40f3cc16910cb7d5eabe"
Oct 31 01:53:04.469704 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.469586    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aad3384e27760598dc81cc9262ec53bf8236bcd4a61d40f3cc16910cb7d5eabe"} err="failed to get container status \"aad3384e27760598dc81cc9262ec53bf8236bcd4a61d40f3cc16910cb7d5eabe\": rpc error: code = NotFound desc = could not find container \"aad3384e27760598dc81cc9262ec53bf8236bcd4a61d40f3cc16910cb7d5eabe\": container with ID starting with aad3384e27760598dc81cc9262ec53bf8236bcd4a61d40f3cc16910cb7d5eabe not found: ID does not exist"
Oct 31 01:53:04.469704 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.469612    5650 scope.go:122] "RemoveContainer" containerID="5a21a20c2f28ed5c46d99e3b0116a73d83902da91ae618de611cf55bbab1f83a"
Oct 31 01:53:04.493207 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.493164    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft"]
Oct 31 01:53:04.503427 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.503320    5650 scope.go:122] "RemoveContainer" containerID="5a21a20c2f28ed5c46d99e3b0116a73d83902da91ae618de611cf55bbab1f83a"
Oct 31 01:53:04.504036 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.504009    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-278a56-786645f87f-jdwft"]
Oct 31 01:53:04.504199 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:53:04.504171    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a21a20c2f28ed5c46d99e3b0116a73d83902da91ae618de611cf55bbab1f83a\": container with ID starting with 5a21a20c2f28ed5c46d99e3b0116a73d83902da91ae618de611cf55bbab1f83a not found: ID does not exist" containerID="5a21a20c2f28ed5c46d99e3b0116a73d83902da91ae618de611cf55bbab1f83a"
Oct 31 01:53:04.504271 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.504213    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a21a20c2f28ed5c46d99e3b0116a73d83902da91ae618de611cf55bbab1f83a"} err="failed to get container status \"5a21a20c2f28ed5c46d99e3b0116a73d83902da91ae618de611cf55bbab1f83a\": rpc error: code = NotFound desc = could not find container \"5a21a20c2f28ed5c46d99e3b0116a73d83902da91ae618de611cf55bbab1f83a\": container with ID starting with 5a21a20c2f28ed5c46d99e3b0116a73d83902da91ae618de611cf55bbab1f83a not found: ID does not exist"
Oct 31 01:53:04.517836 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.517776    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf"]
Oct 31 01:53:04.523055 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.523016    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-278a56-b55dfbf6c-wl4xf"]
Oct 31 01:53:04.536140 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.536107    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22a469a4-b8b4-4601-a954-a992c76be571-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:04.536140 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.536141    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltj7s\" (UniqueName: \"kubernetes.io/projected/22a469a4-b8b4-4601-a954-a992c76be571-kube-api-access-ltj7s\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:04.712718 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.712684    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"]
Oct 31 01:53:04.713317 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.713302    5650 cpu_manager.go:414] "containerMap: removing container" podUID="22a469a4-b8b4-4601-a954-a992c76be571" containerName="payload-pre-processing"
Oct 31 01:53:04.713361 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.713318    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a469a4-b8b4-4601-a954-a992c76be571" containerName="payload-pre-processing"
Oct 31 01:53:04.713361 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.713329    5650 cpu_manager.go:414] "containerMap: removing container" podUID="786fd80e-b891-4ccc-a776-069b6f6ba3f9" containerName="payload-processing"
Oct 31 01:53:04.713361 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.713334    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="786fd80e-b891-4ccc-a776-069b6f6ba3f9" containerName="payload-processing"
Oct 31 01:53:04.713361 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.713344    5650 cpu_manager.go:414] "containerMap: removing container" podUID="fe46844f-a0d6-4210-8516-f4facfbe165f" containerName="payload-processing"
Oct 31 01:53:04.713361 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.713349    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe46844f-a0d6-4210-8516-f4facfbe165f" containerName="payload-processing"
Oct 31 01:53:04.713540 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.713368    5650 cpu_manager.go:414] "containerMap: removing container" podUID="840dad17-e4e9-4a19-83d4-af8ce9ea1aad" containerName="payload-pre-processing"
Oct 31 01:53:04.713540 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.713373    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="840dad17-e4e9-4a19-83d4-af8ce9ea1aad" containerName="payload-pre-processing"
Oct 31 01:53:04.713540 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.713497    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a469a4-b8b4-4601-a954-a992c76be571" containerName="payload-pre-processing"
Oct 31 01:53:04.713540 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.713507    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="786fd80e-b891-4ccc-a776-069b6f6ba3f9" containerName="payload-processing"
Oct 31 01:53:04.713540 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.713514    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe46844f-a0d6-4210-8516-f4facfbe165f" containerName="payload-processing"
Oct 31 01:53:04.713540 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.713524    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="840dad17-e4e9-4a19-83d4-af8ce9ea1aad" containerName="payload-pre-processing"
Oct 31 01:53:04.718692 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.718667    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.727862 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.727836    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-5f65e1-openshift-default-dockercfg-6wtq5\""
Oct 31 01:53:04.732733 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.732698    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"]
Oct 31 01:53:04.841727 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.841682    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-envoy\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.841909 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.841818    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-token\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.841909 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.841893    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-workload-socket\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.842059 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.842035    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-credential-socket\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.842105 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.842084    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istiod-ca-cert\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.842175 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.842162    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-data\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.842244 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.842231    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-podinfo\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.842278 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.842270    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhtn7\" (UniqueName: \"kubernetes.io/projected/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-kube-api-access-mhtn7\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.842368 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.842355    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-workload-certs\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.952527 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.944917    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-podinfo\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.952527 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.944998    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhtn7\" (UniqueName: \"kubernetes.io/projected/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-kube-api-access-mhtn7\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.952527 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.945086    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-workload-certs\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.952527 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.945171    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-envoy\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.952527 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.945306    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-token\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.952527 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.945364    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-workload-socket\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.952527 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.945574    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-credential-socket\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.952527 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.945630    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istiod-ca-cert\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.952527 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.945689    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-data\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.952527 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.946168    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-data\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.968532 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.956547    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-workload-certs\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.968532 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.957184    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-credential-socket\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.968532 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.957471    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-workload-socket\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.968532 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.958169    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istiod-ca-cert\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.976517 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.975410    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhtn7\" (UniqueName: \"kubernetes.io/projected/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-kube-api-access-mhtn7\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.980996 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.980642    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-token\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.994164 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.993488    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-podinfo\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:04.997264 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:04.997234    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-envoy\") pod \"e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") " pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:05.055686 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.055595    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:05.288592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.288512    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"]
Oct 31 01:53:05.294924 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.294895    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.303924 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.303895    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-925bac-openshift-default-dockercfg-tr4kp\""
Oct 31 01:53:05.313227 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.313194    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"]
Oct 31 01:53:05.394229 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.394205    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"]
Oct 31 01:53:05.397764 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:53:05.397722    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd2bc78_97f2_4946_9a5d_f4952c4b49c4.slice/crio-faab513cb79265896f1d893f6ebaf97274720a41775f734cba1486cdaca93c23 WatchSource:0}: Error finding container faab513cb79265896f1d893f6ebaf97274720a41775f734cba1486cdaca93c23: Status 404 returned error can't find the container with id faab513cb79265896f1d893f6ebaf97274720a41775f734cba1486cdaca93c23
Oct 31 01:53:05.400559 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.400526    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:53:05.400677 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.400611    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:53:05.400677 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.400657    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:53:05.456688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.456655    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shqp8\" (UniqueName: \"kubernetes.io/projected/4519a56f-4611-4338-aa8c-aad734828add-kube-api-access-shqp8\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.456854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.456723    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-workload-socket\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.456919 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.456864    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-workload-certs\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.456919 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.456899    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-credential-socket\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.457025 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.456921    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4519a56f-4611-4338-aa8c-aad734828add-istio-token\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.457080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.457027    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-istio-envoy\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.457144 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.457126    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-istio-data\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.457221 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.457207    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4519a56f-4611-4338-aa8c-aad734828add-istiod-ca-cert\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.457283 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.457268    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4519a56f-4611-4338-aa8c-aad734828add-istio-podinfo\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.462203 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.462176    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx" event={"ID":"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4","Type":"ContainerStarted","Data":"faab513cb79265896f1d893f6ebaf97274720a41775f734cba1486cdaca93c23"}
Oct 31 01:53:05.559127 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.559025    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shqp8\" (UniqueName: \"kubernetes.io/projected/4519a56f-4611-4338-aa8c-aad734828add-kube-api-access-shqp8\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.559127 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.559120    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-workload-socket\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.559359 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.559217    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-workload-certs\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.559359 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.559247    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-credential-socket\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.559359 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.559275    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4519a56f-4611-4338-aa8c-aad734828add-istio-token\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.559359 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.559313    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-istio-envoy\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.559600 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.559368    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-istio-data\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.559600 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.559481    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4519a56f-4611-4338-aa8c-aad734828add-istiod-ca-cert\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.559600 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.559580    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-workload-socket\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.559825 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.559787    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-workload-certs\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.559901 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.559878    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-credential-socket\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.559966 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.559921    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4519a56f-4611-4338-aa8c-aad734828add-istio-podinfo\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.560293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.560267    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4519a56f-4611-4338-aa8c-aad734828add-istiod-ca-cert\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.560557 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.560522    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-istio-data\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.562766 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.562738    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-istio-envoy\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.562986 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.562968    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4519a56f-4611-4338-aa8c-aad734828add-istio-podinfo\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.578232 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.578199    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4519a56f-4611-4338-aa8c-aad734828add-istio-token\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.578388 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.578284    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shqp8\" (UniqueName: \"kubernetes.io/projected/4519a56f-4611-4338-aa8c-aad734828add-kube-api-access-shqp8\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.619253 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.619215    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:05.916876 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.916841    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"]
Oct 31 01:53:05.917060 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:53:05.917028    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4519a56f_4611_4338_aa8c_aad734828add.slice/crio-7098b66c5351e1ae0fc91ca206b09c9aa89c001a5c449b9ee268a938787df534 WatchSource:0}: Error finding container 7098b66c5351e1ae0fc91ca206b09c9aa89c001a5c449b9ee268a938787df534: Status 404 returned error can't find the container with id 7098b66c5351e1ae0fc91ca206b09c9aa89c001a5c449b9ee268a938787df534
Oct 31 01:53:05.920002 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.919967    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:53:05.920094 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.920037    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:53:05.920094 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:05.920064    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:53:06.121933 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:06.121897    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22a469a4-b8b4-4601-a954-a992c76be571" path="/var/lib/kubelet/pods/22a469a4-b8b4-4601-a954-a992c76be571/volumes"
Oct 31 01:53:06.122696 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:06.122676    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="786fd80e-b891-4ccc-a776-069b6f6ba3f9" path="/var/lib/kubelet/pods/786fd80e-b891-4ccc-a776-069b6f6ba3f9/volumes"
Oct 31 01:53:06.471946 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:06.471907    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx" event={"ID":"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4","Type":"ContainerStarted","Data":"4b53119a1a231d0eaad1f41d77afdbe6ec0e779fd49ec6cbd7648008aeecdb8a"}
Oct 31 01:53:06.473718 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:06.473686    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f" event={"ID":"4519a56f-4611-4338-aa8c-aad734828add","Type":"ContainerStarted","Data":"8b596584859f5e5149231fde325164683182ea8e3ddd44d45806bfbdffd1d16a"}
Oct 31 01:53:06.473851 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:06.473736    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f" event={"ID":"4519a56f-4611-4338-aa8c-aad734828add","Type":"ContainerStarted","Data":"7098b66c5351e1ae0fc91ca206b09c9aa89c001a5c449b9ee268a938787df534"}
Oct 31 01:53:06.500620 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:06.500558    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx" podStartSLOduration=2.50054426 podStartE2EDuration="2.50054426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:06.497312027 +0000 UTC m=+1942.501026327" watchObservedRunningTime="2026-10-31 01:53:06.50054426 +0000 UTC m=+1942.504258620"
Oct 31 01:53:06.529182 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:06.529119    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f" podStartSLOduration=1.5291039039999998 podStartE2EDuration="1.529103904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:06.525739761 +0000 UTC m=+1942.529454053" watchObservedRunningTime="2026-10-31 01:53:06.529103904 +0000 UTC m=+1942.532818195"
Oct 31 01:53:06.619787 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:06.619751    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:06.621387 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:06.621360    5650 patch_prober.go:29] interesting pod/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Oct 31 01:53:06.621539 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:06.621411    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f" podUID="4519a56f-4611-4338-aa8c-aad734828add" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Oct 31 01:53:07.056071 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:07.056024    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:07.062053 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:07.062025    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:07.336987 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:07.336899    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd"
Oct 31 01:53:07.338689 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:07.338662    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5"
Oct 31 01:53:07.483930 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:07.483879    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:07.484934 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:07.484908    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:07.626522 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:07.624924    5650 patch_prober.go:29] interesting pod/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Oct 31 01:53:07.626522 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:07.624988    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f" podUID="4519a56f-4611-4338-aa8c-aad734828add" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Oct 31 01:53:08.625836 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:08.625800    5650 patch_prober.go:29] interesting pod/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Oct 31 01:53:08.626525 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:08.626486    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f" podUID="4519a56f-4611-4338-aa8c-aad734828add" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Oct 31 01:53:09.634477 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:09.623010    5650 patch_prober.go:29] interesting pod/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Oct 31 01:53:09.634477 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:09.623073    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f" podUID="4519a56f-4611-4338-aa8c-aad734828add" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Oct 31 01:53:10.624953 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:10.624503    5650 patch_prober.go:29] interesting pod/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Oct 31 01:53:10.624953 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:10.624575    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f" podUID="4519a56f-4611-4338-aa8c-aad734828add" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Oct 31 01:53:11.134124 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:11.134079    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828"]
Oct 31 01:53:11.627469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:11.623938    5650 patch_prober.go:29] interesting pod/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Oct 31 01:53:11.627469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:11.624035    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f" podUID="4519a56f-4611-4338-aa8c-aad734828add" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Oct 31 01:53:12.620168 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:12.620122    5650 patch_prober.go:29] interesting pod/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Oct 31 01:53:12.620633 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:12.620211    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f" podUID="4519a56f-4611-4338-aa8c-aad734828add" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Oct 31 01:53:13.600359 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:13.600330    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-lq6ms_d5f6f417-2dd1-4f4e-ad96-92d1f512cac1/payload-processing/0.log"
Oct 31 01:53:13.624256 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:13.624215    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:13.624972 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:13.624806    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:13.625645 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:13.625622    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:53:13.827696 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:13.827665    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-lq6ms_d5f6f417-2dd1-4f4e-ad96-92d1f512cac1/payload-processing/0.log"
Oct 31 01:53:14.024884 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:14.024727    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5_be03a565-244e-4052-a13c-97bf29318774/payload-processing/0.log"
Oct 31 01:53:16.134478 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:16.134393    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828" podUID="3e756d5a-ab71-453f-853a-2884bfe27fac" containerName="payload-pre-processing" containerID="cri-o://8926b92452cc4bf50e48e06b15422719840b4938ec31dac479aece0bf67478b9" gracePeriod=25
Oct 31 01:53:16.573648 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:16.573608    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828" podUID="3e756d5a-ab71-453f-853a-2884bfe27fac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:53:20.036340 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:20.036299    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f488f08-kr8ds"]
Oct 31 01:53:20.040899 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:20.040879    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f488f08-kr8ds"
Oct 31 01:53:20.046945 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:20.046914    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:53:20.047535 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:20.047513    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:53:20.049053 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:20.049026    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f488f08-kr8ds"]
Oct 31 01:53:20.174028 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:20.173952    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzf7w\" (UniqueName: \"kubernetes.io/projected/16b1ef53-8ed9-4180-9086-e60ce2c50dc7-kube-api-access-pzf7w\") pod \"maas-api-revoke-sub-0f488f08-kr8ds\" (UID: \"16b1ef53-8ed9-4180-9086-e60ce2c50dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f488f08-kr8ds"
Oct 31 01:53:20.174245 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:20.174178    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16b1ef53-8ed9-4180-9086-e60ce2c50dc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f488f08-kr8ds\" (UID: \"16b1ef53-8ed9-4180-9086-e60ce2c50dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f488f08-kr8ds"
Oct 31 01:53:20.174397 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:20.174374    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16b1ef53-8ed9-4180-9086-e60ce2c50dc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f488f08-kr8ds\" (UID: \"16b1ef53-8ed9-4180-9086-e60ce2c50dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f488f08-kr8ds"
Oct 31 01:53:20.276427 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:20.276386    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16b1ef53-8ed9-4180-9086-e60ce2c50dc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f488f08-kr8ds\" (UID: \"16b1ef53-8ed9-4180-9086-e60ce2c50dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f488f08-kr8ds"
Oct 31 01:53:20.276662 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:20.276545    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzf7w\" (UniqueName: \"kubernetes.io/projected/16b1ef53-8ed9-4180-9086-e60ce2c50dc7-kube-api-access-pzf7w\") pod \"maas-api-revoke-sub-0f488f08-kr8ds\" (UID: \"16b1ef53-8ed9-4180-9086-e60ce2c50dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f488f08-kr8ds"
Oct 31 01:53:20.276662 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:20.276624    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16b1ef53-8ed9-4180-9086-e60ce2c50dc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f488f08-kr8ds\" (UID: \"16b1ef53-8ed9-4180-9086-e60ce2c50dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f488f08-kr8ds"
Oct 31 01:53:20.277657 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:20.277583    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16b1ef53-8ed9-4180-9086-e60ce2c50dc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f488f08-kr8ds\" (UID: \"16b1ef53-8ed9-4180-9086-e60ce2c50dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f488f08-kr8ds"
Oct 31 01:53:20.280032 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:20.280006    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16b1ef53-8ed9-4180-9086-e60ce2c50dc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f488f08-kr8ds\" (UID: \"16b1ef53-8ed9-4180-9086-e60ce2c50dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f488f08-kr8ds"
Oct 31 01:53:20.308423 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:20.308112    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzf7w\" (UniqueName: \"kubernetes.io/projected/16b1ef53-8ed9-4180-9086-e60ce2c50dc7-kube-api-access-pzf7w\") pod \"maas-api-revoke-sub-0f488f08-kr8ds\" (UID: \"16b1ef53-8ed9-4180-9086-e60ce2c50dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f488f08-kr8ds"
Oct 31 01:53:20.362233 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:20.362192    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f488f08-kr8ds"
Oct 31 01:53:20.660711 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:20.660683    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f488f08-kr8ds"]
Oct 31 01:53:20.661773 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:53:20.661747    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b1ef53_8ed9_4180_9086_e60ce2c50dc7.slice/crio-e1a277427a0a4f6553991d65d1eb1987553fd443600ea96062f20bc79fa3e481 WatchSource:0}: Error finding container e1a277427a0a4f6553991d65d1eb1987553fd443600ea96062f20bc79fa3e481: Status 404 returned error can't find the container with id e1a277427a0a4f6553991d65d1eb1987553fd443600ea96062f20bc79fa3e481
Oct 31 01:53:20.835266 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:20.835231    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f488f08-kr8ds" event={"ID":"16b1ef53-8ed9-4180-9086-e60ce2c50dc7","Type":"ContainerStarted","Data":"e1a277427a0a4f6553991d65d1eb1987553fd443600ea96062f20bc79fa3e481"}
Oct 31 01:53:21.844327 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:21.844286    5650 generic.go:356] "Generic (PLEG): container finished" podID="16b1ef53-8ed9-4180-9086-e60ce2c50dc7" containerID="11707442b47e158cb31d6558bd08a3de40796b4ddceff9c404d7be1f57bb321e" exitCode=0
Oct 31 01:53:21.844734 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:21.844366    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f488f08-kr8ds" event={"ID":"16b1ef53-8ed9-4180-9086-e60ce2c50dc7","Type":"ContainerDied","Data":"11707442b47e158cb31d6558bd08a3de40796b4ddceff9c404d7be1f57bb321e"}
Oct 31 01:53:23.052794 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:23.052765    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f488f08-kr8ds"
Oct 31 01:53:23.120263 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:23.120166    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16b1ef53-8ed9-4180-9086-e60ce2c50dc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16b1ef53-8ed9-4180-9086-e60ce2c50dc7-maas-api-openshift-ca\") pod \"16b1ef53-8ed9-4180-9086-e60ce2c50dc7\" (UID: \"16b1ef53-8ed9-4180-9086-e60ce2c50dc7\") "
Oct 31 01:53:23.120497 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:23.120428    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16b1ef53-8ed9-4180-9086-e60ce2c50dc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16b1ef53-8ed9-4180-9086-e60ce2c50dc7-maas-api-xks-ca\") pod \"16b1ef53-8ed9-4180-9086-e60ce2c50dc7\" (UID: \"16b1ef53-8ed9-4180-9086-e60ce2c50dc7\") "
Oct 31 01:53:23.120632 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:23.120606    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b1ef53-8ed9-4180-9086-e60ce2c50dc7-maas-api-openshift-ca" pod "16b1ef53-8ed9-4180-9086-e60ce2c50dc7" (UID: "16b1ef53-8ed9-4180-9086-e60ce2c50dc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:23.120696 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:23.120670    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b1ef53-8ed9-4180-9086-e60ce2c50dc7-kube-api-access-pzf7w\" (UniqueName: \"kubernetes.io/projected/16b1ef53-8ed9-4180-9086-e60ce2c50dc7-kube-api-access-pzf7w\") pod \"16b1ef53-8ed9-4180-9086-e60ce2c50dc7\" (UID: \"16b1ef53-8ed9-4180-9086-e60ce2c50dc7\") "
Oct 31 01:53:23.121256 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:23.121231    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16b1ef53-8ed9-4180-9086-e60ce2c50dc7-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:23.122915 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:23.122884    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16b1ef53-8ed9-4180-9086-e60ce2c50dc7-maas-api-xks-ca" pod "16b1ef53-8ed9-4180-9086-e60ce2c50dc7" (UID: "16b1ef53-8ed9-4180-9086-e60ce2c50dc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:53:23.124706 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:23.124678    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b1ef53-8ed9-4180-9086-e60ce2c50dc7-kube-api-access-pzf7w" pod "16b1ef53-8ed9-4180-9086-e60ce2c50dc7" (UID: "16b1ef53-8ed9-4180-9086-e60ce2c50dc7"). InnerVolumeSpecName "kube-api-access-pzf7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:23.223012 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:23.222980    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16b1ef53-8ed9-4180-9086-e60ce2c50dc7-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:23.223012 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:23.223009    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzf7w\" (UniqueName: \"kubernetes.io/projected/16b1ef53-8ed9-4180-9086-e60ce2c50dc7-kube-api-access-pzf7w\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:23.862903 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:23.862874    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f488f08-kr8ds"
Oct 31 01:53:23.863118 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:23.862918    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f488f08-kr8ds" event={"ID":"16b1ef53-8ed9-4180-9086-e60ce2c50dc7","Type":"ContainerDied","Data":"e1a277427a0a4f6553991d65d1eb1987553fd443600ea96062f20bc79fa3e481"}
Oct 31 01:53:23.863118 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:23.862969    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1a277427a0a4f6553991d65d1eb1987553fd443600ea96062f20bc79fa3e481"
Oct 31 01:53:26.573387 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:26.573348    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828" podUID="3e756d5a-ab71-453f-853a-2884bfe27fac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:53:28.555029 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:53:28.554989    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:53:28.945300 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:28.945166    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt"]
Oct 31 01:53:28.945495 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:28.945429    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt" podUID="548329c1-14a7-4efe-844f-248210658f41" containerName="maas-api" containerID="cri-o://7c40721e028a9baa4a23bdb0e0e3bf8dc7ad203e77c7faa016b4555fbddd25a4" gracePeriod=30
Oct 31 01:53:29.258582 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:29.258551    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt"
Oct 31 01:53:29.334237 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:29.334203    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/548329c1-14a7-4efe-844f-248210658f41-metrics-certs\" (UniqueName: \"kubernetes.io/secret/548329c1-14a7-4efe-844f-248210658f41-metrics-certs\") pod \"548329c1-14a7-4efe-844f-248210658f41\" (UID: \"548329c1-14a7-4efe-844f-248210658f41\") "
Oct 31 01:53:29.334471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:29.334370    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/548329c1-14a7-4efe-844f-248210658f41-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/548329c1-14a7-4efe-844f-248210658f41-openshift-service-ca\") pod \"548329c1-14a7-4efe-844f-248210658f41\" (UID: \"548329c1-14a7-4efe-844f-248210658f41\") "
Oct 31 01:53:29.334563 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:29.334485    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/548329c1-14a7-4efe-844f-248210658f41-kube-api-access-ft6d4\" (UniqueName: \"kubernetes.io/projected/548329c1-14a7-4efe-844f-248210658f41-kube-api-access-ft6d4\") pod \"548329c1-14a7-4efe-844f-248210658f41\" (UID: \"548329c1-14a7-4efe-844f-248210658f41\") "
Oct 31 01:53:29.334563 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:29.334514    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/548329c1-14a7-4efe-844f-248210658f41-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/548329c1-14a7-4efe-844f-248210658f41-maas-api-tls\") pod \"548329c1-14a7-4efe-844f-248210658f41\" (UID: \"548329c1-14a7-4efe-844f-248210658f41\") "
Oct 31 01:53:29.334845 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:29.334813    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/548329c1-14a7-4efe-844f-248210658f41-openshift-service-ca" pod "548329c1-14a7-4efe-844f-248210658f41" (UID: "548329c1-14a7-4efe-844f-248210658f41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:29.335216 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:29.335191    5650 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/548329c1-14a7-4efe-844f-248210658f41-openshift-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:29.337006 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:29.336979    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/548329c1-14a7-4efe-844f-248210658f41-maas-api-tls" pod "548329c1-14a7-4efe-844f-248210658f41" (UID: "548329c1-14a7-4efe-844f-248210658f41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:53:29.337213 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:29.337188    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/548329c1-14a7-4efe-844f-248210658f41-metrics-certs" pod "548329c1-14a7-4efe-844f-248210658f41" (UID: "548329c1-14a7-4efe-844f-248210658f41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:53:29.337413 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:29.337385    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/548329c1-14a7-4efe-844f-248210658f41-kube-api-access-ft6d4" pod "548329c1-14a7-4efe-844f-248210658f41" (UID: "548329c1-14a7-4efe-844f-248210658f41"). InnerVolumeSpecName "kube-api-access-ft6d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:29.482524 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:29.468206    5650 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/548329c1-14a7-4efe-844f-248210658f41-metrics-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:29.482524 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:29.468250    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft6d4\" (UniqueName: \"kubernetes.io/projected/548329c1-14a7-4efe-844f-248210658f41-kube-api-access-ft6d4\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:29.482524 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:29.468266    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/548329c1-14a7-4efe-844f-248210658f41-maas-api-tls\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:29.913562 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:29.913441    5650 generic.go:356] "Generic (PLEG): container finished" podID="548329c1-14a7-4efe-844f-248210658f41" containerID="7c40721e028a9baa4a23bdb0e0e3bf8dc7ad203e77c7faa016b4555fbddd25a4" exitCode=0
Oct 31 01:53:29.913562 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:29.913500    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt" event={"ID":"548329c1-14a7-4efe-844f-248210658f41","Type":"ContainerDied","Data":"7c40721e028a9baa4a23bdb0e0e3bf8dc7ad203e77c7faa016b4555fbddd25a4"}
Oct 31 01:53:29.913562 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:29.913527    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt"
Oct 31 01:53:29.913562 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:29.913557    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt" event={"ID":"548329c1-14a7-4efe-844f-248210658f41","Type":"ContainerDied","Data":"90e4cdfcaf0940a2fd670d85adc9ab541a8a739caa5d3b3f26a4cadedfcfb36a"}
Oct 31 01:53:29.914076 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:29.913584    5650 scope.go:122] "RemoveContainer" containerID="7c40721e028a9baa4a23bdb0e0e3bf8dc7ad203e77c7faa016b4555fbddd25a4"
Oct 31 01:53:29.925231 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:29.925208    5650 scope.go:122] "RemoveContainer" containerID="7c40721e028a9baa4a23bdb0e0e3bf8dc7ad203e77c7faa016b4555fbddd25a4"
Oct 31 01:53:29.925555 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:53:29.925533    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c40721e028a9baa4a23bdb0e0e3bf8dc7ad203e77c7faa016b4555fbddd25a4\": container with ID starting with 7c40721e028a9baa4a23bdb0e0e3bf8dc7ad203e77c7faa016b4555fbddd25a4 not found: ID does not exist" containerID="7c40721e028a9baa4a23bdb0e0e3bf8dc7ad203e77c7faa016b4555fbddd25a4"
Oct 31 01:53:29.925633 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:29.925564    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c40721e028a9baa4a23bdb0e0e3bf8dc7ad203e77c7faa016b4555fbddd25a4"} err="failed to get container status \"7c40721e028a9baa4a23bdb0e0e3bf8dc7ad203e77c7faa016b4555fbddd25a4\": rpc error: code = NotFound desc = could not find container \"7c40721e028a9baa4a23bdb0e0e3bf8dc7ad203e77c7faa016b4555fbddd25a4\": container with ID starting with 7c40721e028a9baa4a23bdb0e0e3bf8dc7ad203e77c7faa016b4555fbddd25a4 not found: ID does not exist"
Oct 31 01:53:29.942828 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:29.942797    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt"]
Oct 31 01:53:29.948803 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:29.948777    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a3ff0c-7c6df7f677-t5tgt"]
Oct 31 01:53:30.119646 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:30.119611    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="548329c1-14a7-4efe-844f-248210658f41" path="/var/lib/kubelet/pods/548329c1-14a7-4efe-844f-248210658f41/volumes"
Oct 31 01:53:33.334591 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:33.334533    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:33.335331 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:33.335197    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:33.367204 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:33.366603    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:33.381591 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:33.381544    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:33.382253 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:33.382218    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:33.960812 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:33.958090    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 01:53:34.065506 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:34.060329    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded"
Oct 31 01:53:34.182467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:34.175945    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:34.291551 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:34.289564    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:34.338472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:34.338088    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:34.343471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:34.339381    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:34.343471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:34.339756    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:53:34.372471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:34.367046    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:34.372471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:34.367476    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:53:34.372471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:34.367470    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 01:53:34.372471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:34.368460    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:34.372471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:34.368895    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:53:34.385471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:34.382782    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:34.389468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:34.388740    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:53:34.565542 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:34.517097    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:53:34.622767 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:34.617909    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:53:34.778961 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:34.768960    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:34.971653 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:34.968543    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:34.971653 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:34.969750    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:53:35.025044 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:35.002523    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:35.317524 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:35.307015    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:35.317524 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:35.307676    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:53:35.371365 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:35.371315    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:35.373941 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:35.373904    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 01:53:35.380348 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:35.380314    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:53:35.412213 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:35.412172    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 01:53:35.414496 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:35.414298    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:53:35.442742 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:35.434556    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:35.484546 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:35.484335    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:53:35.491082 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:35.491050    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:53:35.523149 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:35.523102    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:35.958465 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:35.955177    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:53:35.958465 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:35.957684    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:53:35.968760 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:35.968714    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:35.985634 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:35.985602    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:53:35.987528 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:35.987507    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:53:36.448194 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:36.448156    5650 generic.go:356] "Generic (PLEG): container finished" podID="3e756d5a-ab71-453f-853a-2884bfe27fac" containerID="8926b92452cc4bf50e48e06b15422719840b4938ec31dac479aece0bf67478b9" exitCode=0
Oct 31 01:53:36.450421 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:36.449196    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828" event={"ID":"3e756d5a-ab71-453f-853a-2884bfe27fac","Type":"ContainerDied","Data":"8926b92452cc4bf50e48e06b15422719840b4938ec31dac479aece0bf67478b9"}
Oct 31 01:53:36.495907 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:36.495876    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828"
Oct 31 01:53:36.598399 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:36.598305    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e756d5a-ab71-453f-853a-2884bfe27fac-kube-api-access-l4zkm\" (UniqueName: \"kubernetes.io/projected/3e756d5a-ab71-453f-853a-2884bfe27fac-kube-api-access-l4zkm\") pod \"3e756d5a-ab71-453f-853a-2884bfe27fac\" (UID: \"3e756d5a-ab71-453f-853a-2884bfe27fac\") "
Oct 31 01:53:36.598628 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:36.598413    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e756d5a-ab71-453f-853a-2884bfe27fac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e756d5a-ab71-453f-853a-2884bfe27fac-plugins-config-volume\") pod \"3e756d5a-ab71-453f-853a-2884bfe27fac\" (UID: \"3e756d5a-ab71-453f-853a-2884bfe27fac\") "
Oct 31 01:53:36.598807 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:36.598782    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e756d5a-ab71-453f-853a-2884bfe27fac-plugins-config-volume" pod "3e756d5a-ab71-453f-853a-2884bfe27fac" (UID: "3e756d5a-ab71-453f-853a-2884bfe27fac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:36.598956 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:36.598943    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e756d5a-ab71-453f-853a-2884bfe27fac-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:36.625849 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:36.625411    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e756d5a-ab71-453f-853a-2884bfe27fac-kube-api-access-l4zkm" pod "3e756d5a-ab71-453f-853a-2884bfe27fac" (UID: "3e756d5a-ab71-453f-853a-2884bfe27fac"). InnerVolumeSpecName "kube-api-access-l4zkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:36.724528 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:36.723723    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4zkm\" (UniqueName: \"kubernetes.io/projected/3e756d5a-ab71-453f-853a-2884bfe27fac-kube-api-access-l4zkm\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:37.461006 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:37.460963    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828" event={"ID":"3e756d5a-ab71-453f-853a-2884bfe27fac","Type":"ContainerDied","Data":"64f434ad7f3e1742587b59204d365ee6df34a8a2fa93b7ccd8d24acf6a5304c5"}
Oct 31 01:53:37.461690 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:37.461044    5650 scope.go:122] "RemoveContainer" containerID="8926b92452cc4bf50e48e06b15422719840b4938ec31dac479aece0bf67478b9"
Oct 31 01:53:37.461690 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:37.461244    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828"
Oct 31 01:53:37.499128 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:37.499089    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828"]
Oct 31 01:53:37.531333 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:37.531294    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a779b3-5d8fff9cc9-pn828"]
Oct 31 01:53:37.997495 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:37.996419    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"]
Oct 31 01:53:37.997495 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:37.996718    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq" podUID="eed0a2f2-6812-4d37-b175-7ca1d14d5f9c" containerName="istio-proxy" containerID="cri-o://fcf4c4dcc2fd7b674b3ee7c0e2ad9e14b8283c0211760399130dbcbf0285427b" gracePeriod=30
Oct 31 01:53:38.228466 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:38.228220    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e756d5a-ab71-453f-853a-2884bfe27fac" path="/var/lib/kubelet/pods/3e756d5a-ab71-453f-853a-2884bfe27fac/volumes"
Oct 31 01:53:43.291106 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.291077    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:53:43.361896 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.361852    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-podinfo\") pod \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") "
Oct 31 01:53:43.362089 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.361949    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-data\") pod \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") "
Oct 31 01:53:43.362089 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.361984    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-workload-socket\") pod \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") "
Oct 31 01:53:43.362089 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.362043    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-workload-certs\") pod \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") "
Oct 31 01:53:43.362269 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.362089    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-kube-api-access-vf9p9\" (UniqueName: \"kubernetes.io/projected/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-kube-api-access-vf9p9\") pod \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") "
Oct 31 01:53:43.362329 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.362266    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-token\" (UniqueName: \"kubernetes.io/projected/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-token\") pod \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") "
Oct 31 01:53:43.362329 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.362306    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istiod-ca-cert\") pod \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") "
Oct 31 01:53:43.362460 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.362325    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-data" pod "eed0a2f2-6812-4d37-b175-7ca1d14d5f9c" (UID: "eed0a2f2-6812-4d37-b175-7ca1d14d5f9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:53:43.362460 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.362361    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-workload-socket" pod "eed0a2f2-6812-4d37-b175-7ca1d14d5f9c" (UID: "eed0a2f2-6812-4d37-b175-7ca1d14d5f9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:53:43.362460 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.362387    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-credential-socket\") pod \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") "
Oct 31 01:53:43.362621 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.362520    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-envoy\") pod \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\" (UID: \"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c\") "
Oct 31 01:53:43.362708 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.362684    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-workload-certs" pod "eed0a2f2-6812-4d37-b175-7ca1d14d5f9c" (UID: "eed0a2f2-6812-4d37-b175-7ca1d14d5f9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:53:43.362802 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.362778    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-credential-socket" pod "eed0a2f2-6812-4d37-b175-7ca1d14d5f9c" (UID: "eed0a2f2-6812-4d37-b175-7ca1d14d5f9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:53:43.362860 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.362819    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istiod-ca-cert" pod "eed0a2f2-6812-4d37-b175-7ca1d14d5f9c" (UID: "eed0a2f2-6812-4d37-b175-7ca1d14d5f9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:43.363261 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.363240    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:43.363349 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.363268    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:43.363349 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.363284    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:43.363349 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.363297    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:43.363349 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.363312    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:43.365045 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.365018    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-podinfo" pod "eed0a2f2-6812-4d37-b175-7ca1d14d5f9c" (UID: "eed0a2f2-6812-4d37-b175-7ca1d14d5f9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:53:43.365560 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.365527    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-token" pod "eed0a2f2-6812-4d37-b175-7ca1d14d5f9c" (UID: "eed0a2f2-6812-4d37-b175-7ca1d14d5f9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:43.365677 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.365570    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-kube-api-access-vf9p9" pod "eed0a2f2-6812-4d37-b175-7ca1d14d5f9c" (UID: "eed0a2f2-6812-4d37-b175-7ca1d14d5f9c"). InnerVolumeSpecName "kube-api-access-vf9p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:43.366091 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.366064    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-envoy" pod "eed0a2f2-6812-4d37-b175-7ca1d14d5f9c" (UID: "eed0a2f2-6812-4d37-b175-7ca1d14d5f9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:53:43.464740 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.464709    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:43.464740 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.464738    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:43.464740 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.464748    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf9p9\" (UniqueName: \"kubernetes.io/projected/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-kube-api-access-vf9p9\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:43.465314 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.464758    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:43.522335 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.522299    5650 generic.go:356] "Generic (PLEG): container finished" podID="0fd2bc78-97f2-4946-9a5d-f4952c4b49c4" containerID="4b53119a1a231d0eaad1f41d77afdbe6ec0e779fd49ec6cbd7648008aeecdb8a" exitCode=0
Oct 31 01:53:43.522593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.522498    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx" event={"ID":"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4","Type":"ContainerDied","Data":"4b53119a1a231d0eaad1f41d77afdbe6ec0e779fd49ec6cbd7648008aeecdb8a"}
Oct 31 01:53:43.524074 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.523236    5650 scope.go:122] "RemoveContainer" containerID="4b53119a1a231d0eaad1f41d77afdbe6ec0e779fd49ec6cbd7648008aeecdb8a"
Oct 31 01:53:43.530801 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.530764    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:53:43.530945 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.530853    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:53:43.530945 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.530902    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:53:43.531417 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.531392    5650 generic.go:356] "Generic (PLEG): container finished" podID="eed0a2f2-6812-4d37-b175-7ca1d14d5f9c" containerID="fcf4c4dcc2fd7b674b3ee7c0e2ad9e14b8283c0211760399130dbcbf0285427b" exitCode=0
Oct 31 01:53:43.531634 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.531572    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq" event={"ID":"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c","Type":"ContainerDied","Data":"fcf4c4dcc2fd7b674b3ee7c0e2ad9e14b8283c0211760399130dbcbf0285427b"}
Oct 31 01:53:43.531634 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.531626    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq" event={"ID":"eed0a2f2-6812-4d37-b175-7ca1d14d5f9c","Type":"ContainerDied","Data":"3d24aa07291b864fb41501c9d2ebd5e141c01d1bb07c490e7b923123c3dd477c"}
Oct 31 01:53:43.531743 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.531658    5650 scope.go:122] "RemoveContainer" containerID="fcf4c4dcc2fd7b674b3ee7c0e2ad9e14b8283c0211760399130dbcbf0285427b"
Oct 31 01:53:43.531859 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.531808    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"
Oct 31 01:53:43.551930 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.551884    5650 scope.go:122] "RemoveContainer" containerID="af785e4b02c3d66b8906969382a903d60fe512552ec3a80ef4bf3e2667ec82e8"
Oct 31 01:53:43.572712 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.572681    5650 scope.go:122] "RemoveContainer" containerID="fcf4c4dcc2fd7b674b3ee7c0e2ad9e14b8283c0211760399130dbcbf0285427b"
Oct 31 01:53:43.580985 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:53:43.580939    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcf4c4dcc2fd7b674b3ee7c0e2ad9e14b8283c0211760399130dbcbf0285427b\": container with ID starting with fcf4c4dcc2fd7b674b3ee7c0e2ad9e14b8283c0211760399130dbcbf0285427b not found: ID does not exist" containerID="fcf4c4dcc2fd7b674b3ee7c0e2ad9e14b8283c0211760399130dbcbf0285427b"
Oct 31 01:53:43.581140 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.580988    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcf4c4dcc2fd7b674b3ee7c0e2ad9e14b8283c0211760399130dbcbf0285427b"} err="failed to get container status \"fcf4c4dcc2fd7b674b3ee7c0e2ad9e14b8283c0211760399130dbcbf0285427b\": rpc error: code = NotFound desc = could not find container \"fcf4c4dcc2fd7b674b3ee7c0e2ad9e14b8283c0211760399130dbcbf0285427b\": container with ID starting with fcf4c4dcc2fd7b674b3ee7c0e2ad9e14b8283c0211760399130dbcbf0285427b not found: ID does not exist"
Oct 31 01:53:43.581140 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.581018    5650 scope.go:122] "RemoveContainer" containerID="af785e4b02c3d66b8906969382a903d60fe512552ec3a80ef4bf3e2667ec82e8"
Oct 31 01:53:43.581937 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.581883    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"]
Oct 31 01:53:43.586683 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:53:43.581976    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af785e4b02c3d66b8906969382a903d60fe512552ec3a80ef4bf3e2667ec82e8\": container with ID starting with af785e4b02c3d66b8906969382a903d60fe512552ec3a80ef4bf3e2667ec82e8 not found: ID does not exist" containerID="af785e4b02c3d66b8906969382a903d60fe512552ec3a80ef4bf3e2667ec82e8"
Oct 31 01:53:43.586683 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.583389    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af785e4b02c3d66b8906969382a903d60fe512552ec3a80ef4bf3e2667ec82e8"} err="failed to get container status \"af785e4b02c3d66b8906969382a903d60fe512552ec3a80ef4bf3e2667ec82e8\": rpc error: code = NotFound desc = could not find container \"af785e4b02c3d66b8906969382a903d60fe512552ec3a80ef4bf3e2667ec82e8\": container with ID starting with af785e4b02c3d66b8906969382a903d60fe512552ec3a80ef4bf3e2667ec82e8 not found: ID does not exist"
Oct 31 01:53:43.590280 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:43.590255    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a3ff0c-openshift-default-65b548b59c-sqjlq"]
Oct 31 01:53:44.137402 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.137365    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eed0a2f2-6812-4d37-b175-7ca1d14d5f9c" path="/var/lib/kubelet/pods/eed0a2f2-6812-4d37-b175-7ca1d14d5f9c/volumes"
Oct 31 01:53:44.242253 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.242216    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6407d3cc-dpppj"]
Oct 31 01:53:44.242941 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.242920    5650 cpu_manager.go:414] "containerMap: removing container" podUID="eed0a2f2-6812-4d37-b175-7ca1d14d5f9c" containerName="istio-proxy"
Oct 31 01:53:44.243010 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.242942    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eed0a2f2-6812-4d37-b175-7ca1d14d5f9c" containerName="istio-proxy"
Oct 31 01:53:44.243010 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.242960    5650 cpu_manager.go:414] "containerMap: removing container" podUID="16b1ef53-8ed9-4180-9086-e60ce2c50dc7" containerName="revoke-keys"
Oct 31 01:53:44.243010 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.242966    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b1ef53-8ed9-4180-9086-e60ce2c50dc7" containerName="revoke-keys"
Oct 31 01:53:44.243010 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.242976    5650 cpu_manager.go:414] "containerMap: removing container" podUID="eed0a2f2-6812-4d37-b175-7ca1d14d5f9c" containerName="istio-proxy"
Oct 31 01:53:44.243010 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.242981    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eed0a2f2-6812-4d37-b175-7ca1d14d5f9c" containerName="istio-proxy"
Oct 31 01:53:44.243010 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.243007    5650 cpu_manager.go:414] "containerMap: removing container" podUID="548329c1-14a7-4efe-844f-248210658f41" containerName="maas-api"
Oct 31 01:53:44.243302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.243014    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="548329c1-14a7-4efe-844f-248210658f41" containerName="maas-api"
Oct 31 01:53:44.243302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.243028    5650 cpu_manager.go:414] "containerMap: removing container" podUID="3e756d5a-ab71-453f-853a-2884bfe27fac" containerName="payload-pre-processing"
Oct 31 01:53:44.243302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.243033    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e756d5a-ab71-453f-853a-2884bfe27fac" containerName="payload-pre-processing"
Oct 31 01:53:44.243302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.243132    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="eed0a2f2-6812-4d37-b175-7ca1d14d5f9c" containerName="istio-proxy"
Oct 31 01:53:44.243302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.243142    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b1ef53-8ed9-4180-9086-e60ce2c50dc7" containerName="revoke-keys"
Oct 31 01:53:44.243302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.243148    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="eed0a2f2-6812-4d37-b175-7ca1d14d5f9c" containerName="istio-proxy"
Oct 31 01:53:44.243302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.243159    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="548329c1-14a7-4efe-844f-248210658f41" containerName="maas-api"
Oct 31 01:53:44.243302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.243171    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e756d5a-ab71-453f-853a-2884bfe27fac" containerName="payload-pre-processing"
Oct 31 01:53:44.245807 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.245776    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6407d3cc-dpppj"
Oct 31 01:53:44.256767 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.256746    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:53:44.257129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.257113    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:53:44.272020 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.271982    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6407d3cc-dpppj"]
Oct 31 01:53:44.289737 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.288259    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwg27\" (UniqueName: \"kubernetes.io/projected/21539936-858a-45d4-8dcf-c4e15813af0c-kube-api-access-kwg27\") pod \"maas-api-revoke-sub-6407d3cc-dpppj\" (UID: \"21539936-858a-45d4-8dcf-c4e15813af0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6407d3cc-dpppj"
Oct 31 01:53:44.289737 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.288436    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21539936-858a-45d4-8dcf-c4e15813af0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6407d3cc-dpppj\" (UID: \"21539936-858a-45d4-8dcf-c4e15813af0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6407d3cc-dpppj"
Oct 31 01:53:44.289737 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.288533    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21539936-858a-45d4-8dcf-c4e15813af0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6407d3cc-dpppj\" (UID: \"21539936-858a-45d4-8dcf-c4e15813af0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6407d3cc-dpppj"
Oct 31 01:53:44.390143 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.390055    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwg27\" (UniqueName: \"kubernetes.io/projected/21539936-858a-45d4-8dcf-c4e15813af0c-kube-api-access-kwg27\") pod \"maas-api-revoke-sub-6407d3cc-dpppj\" (UID: \"21539936-858a-45d4-8dcf-c4e15813af0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6407d3cc-dpppj"
Oct 31 01:53:44.390604 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.390218    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21539936-858a-45d4-8dcf-c4e15813af0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6407d3cc-dpppj\" (UID: \"21539936-858a-45d4-8dcf-c4e15813af0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6407d3cc-dpppj"
Oct 31 01:53:44.390604 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.390292    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21539936-858a-45d4-8dcf-c4e15813af0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6407d3cc-dpppj\" (UID: \"21539936-858a-45d4-8dcf-c4e15813af0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6407d3cc-dpppj"
Oct 31 01:53:44.391015 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.390995    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21539936-858a-45d4-8dcf-c4e15813af0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6407d3cc-dpppj\" (UID: \"21539936-858a-45d4-8dcf-c4e15813af0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6407d3cc-dpppj"
Oct 31 01:53:44.392611 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.392592    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21539936-858a-45d4-8dcf-c4e15813af0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6407d3cc-dpppj\" (UID: \"21539936-858a-45d4-8dcf-c4e15813af0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6407d3cc-dpppj"
Oct 31 01:53:44.429171 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.429132    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwg27\" (UniqueName: \"kubernetes.io/projected/21539936-858a-45d4-8dcf-c4e15813af0c-kube-api-access-kwg27\") pod \"maas-api-revoke-sub-6407d3cc-dpppj\" (UID: \"21539936-858a-45d4-8dcf-c4e15813af0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6407d3cc-dpppj"
Oct 31 01:53:44.557147 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.557086    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx" event={"ID":"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4","Type":"ContainerStarted","Data":"0ad9fdc5f00e0e9cd4ed97ef402673832ad221ffc3b9ba4b49b649b556fd560d"}
Oct 31 01:53:44.562655 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.562621    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6407d3cc-dpppj"
Oct 31 01:53:44.944533 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:44.940278    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6407d3cc-dpppj"]
Oct 31 01:53:44.953483 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:53:44.952625    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21539936_858a_45d4_8dcf_c4e15813af0c.slice/crio-9380445128bdfdc0a3b8321b799bc6acba58416c0227b3749fb441d7a7ecc475 WatchSource:0}: Error finding container 9380445128bdfdc0a3b8321b799bc6acba58416c0227b3749fb441d7a7ecc475: Status 404 returned error can't find the container with id 9380445128bdfdc0a3b8321b799bc6acba58416c0227b3749fb441d7a7ecc475
Oct 31 01:53:45.056761 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:45.056724    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:45.057432 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:45.057410    5650 patch_prober.go:29] interesting pod/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 31 01:53:45.057544 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:45.057493    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx" podUID="0fd2bc78-97f2-4946-9a5d-f4952c4b49c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 31 01:53:45.132518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:45.132484    5650 scope.go:122] "RemoveContainer" containerID="a72b91c2a65bf263eae4002e7d1496fda1e1932a04af55ecdf1868289e379346"
Oct 31 01:53:45.567087 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:45.567048    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6407d3cc-dpppj" event={"ID":"21539936-858a-45d4-8dcf-c4e15813af0c","Type":"ContainerStarted","Data":"7e4383e7a682ffd7a5c97308c248bcd606b927a4084d6cf37d14c614d56c9d99"}
Oct 31 01:53:45.567682 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:45.567106    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6407d3cc-dpppj" event={"ID":"21539936-858a-45d4-8dcf-c4e15813af0c","Type":"ContainerStarted","Data":"9380445128bdfdc0a3b8321b799bc6acba58416c0227b3749fb441d7a7ecc475"}
Oct 31 01:53:45.588689 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:45.588631    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6407d3cc-dpppj" podStartSLOduration=1.5886172570000001 podStartE2EDuration="1.588617257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:45.58765635 +0000 UTC m=+1981.591370646" watchObservedRunningTime="2026-10-31 01:53:45.588617257 +0000 UTC m=+1981.592331548"
Oct 31 01:53:46.056153 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:46.056121    5650 patch_prober.go:29] interesting pod/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 31 01:53:46.056335 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:46.056176    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx" podUID="0fd2bc78-97f2-4946-9a5d-f4952c4b49c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 31 01:53:46.395532 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:46.395208    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"]
Oct 31 01:53:46.395704 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:46.395550    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js" podUID="b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f" containerName="istio-proxy" containerID="cri-o://5c7214bc981078f01dbad8bf089d15f5625b711aa105ae8916982c91e212b464" gracePeriod=30
Oct 31 01:53:46.580464 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:46.580419    5650 generic.go:356] "Generic (PLEG): container finished" podID="21539936-858a-45d4-8dcf-c4e15813af0c" containerID="7e4383e7a682ffd7a5c97308c248bcd606b927a4084d6cf37d14c614d56c9d99" exitCode=0
Oct 31 01:53:46.580893 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:46.580509    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6407d3cc-dpppj" event={"ID":"21539936-858a-45d4-8dcf-c4e15813af0c","Type":"ContainerDied","Data":"7e4383e7a682ffd7a5c97308c248bcd606b927a4084d6cf37d14c614d56c9d99"}
Oct 31 01:53:47.056105 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:47.056065    5650 patch_prober.go:29] interesting pod/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 31 01:53:47.056296 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:47.056123    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx" podUID="0fd2bc78-97f2-4946-9a5d-f4952c4b49c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 31 01:53:47.828658 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:47.828627    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6407d3cc-dpppj"
Oct 31 01:53:47.899381 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:47.899272    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21539936-858a-45d4-8dcf-c4e15813af0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21539936-858a-45d4-8dcf-c4e15813af0c-maas-api-openshift-ca\") pod \"21539936-858a-45d4-8dcf-c4e15813af0c\" (UID: \"21539936-858a-45d4-8dcf-c4e15813af0c\") "
Oct 31 01:53:47.899616 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:47.899562    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21539936-858a-45d4-8dcf-c4e15813af0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21539936-858a-45d4-8dcf-c4e15813af0c-maas-api-xks-ca\") pod \"21539936-858a-45d4-8dcf-c4e15813af0c\" (UID: \"21539936-858a-45d4-8dcf-c4e15813af0c\") "
Oct 31 01:53:47.925278 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:47.899730    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21539936-858a-45d4-8dcf-c4e15813af0c-kube-api-access-kwg27\" (UniqueName: \"kubernetes.io/projected/21539936-858a-45d4-8dcf-c4e15813af0c-kube-api-access-kwg27\") pod \"21539936-858a-45d4-8dcf-c4e15813af0c\" (UID: \"21539936-858a-45d4-8dcf-c4e15813af0c\") "
Oct 31 01:53:47.925278 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:47.924677    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21539936-858a-45d4-8dcf-c4e15813af0c-maas-api-openshift-ca" pod "21539936-858a-45d4-8dcf-c4e15813af0c" (UID: "21539936-858a-45d4-8dcf-c4e15813af0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:47.932017 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:47.931984    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21539936-858a-45d4-8dcf-c4e15813af0c-kube-api-access-kwg27" pod "21539936-858a-45d4-8dcf-c4e15813af0c" (UID: "21539936-858a-45d4-8dcf-c4e15813af0c"). InnerVolumeSpecName "kube-api-access-kwg27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:47.933365 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:47.933239    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21539936-858a-45d4-8dcf-c4e15813af0c-maas-api-xks-ca" pod "21539936-858a-45d4-8dcf-c4e15813af0c" (UID: "21539936-858a-45d4-8dcf-c4e15813af0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:53:48.017471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:48.011688    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwg27\" (UniqueName: \"kubernetes.io/projected/21539936-858a-45d4-8dcf-c4e15813af0c-kube-api-access-kwg27\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:48.017471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:48.011730    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21539936-858a-45d4-8dcf-c4e15813af0c-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:48.017471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:48.011746    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21539936-858a-45d4-8dcf-c4e15813af0c-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:48.056871 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:48.056637    5650 patch_prober.go:29] interesting pod/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 31 01:53:48.056871 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:48.056704    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx" podUID="0fd2bc78-97f2-4946-9a5d-f4952c4b49c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 31 01:53:48.618486 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:48.617093    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6407d3cc-dpppj" event={"ID":"21539936-858a-45d4-8dcf-c4e15813af0c","Type":"ContainerDied","Data":"9380445128bdfdc0a3b8321b799bc6acba58416c0227b3749fb441d7a7ecc475"}
Oct 31 01:53:48.618486 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:48.617175    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9380445128bdfdc0a3b8321b799bc6acba58416c0227b3749fb441d7a7ecc475"
Oct 31 01:53:48.618486 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:48.617260    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6407d3cc-dpppj"
Oct 31 01:53:49.056874 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:49.056834    5650 patch_prober.go:29] interesting pod/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 31 01:53:49.057288 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:49.056891    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx" podUID="0fd2bc78-97f2-4946-9a5d-f4952c4b49c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 31 01:53:50.061069 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:50.061034    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:50.061605 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:50.061582    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:50.062307 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:50.062288    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:53:51.656472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.649253    5650 generic.go:356] "Generic (PLEG): container finished" podID="b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f" containerID="5c7214bc981078f01dbad8bf089d15f5625b711aa105ae8916982c91e212b464" exitCode=0
Oct 31 01:53:51.656472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.652209    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js" event={"ID":"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f","Type":"ContainerDied","Data":"5c7214bc981078f01dbad8bf089d15f5625b711aa105ae8916982c91e212b464"}
Oct 31 01:53:51.761929 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.761900    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:53:51.788670 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.788636    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-token\" (UniqueName: \"kubernetes.io/projected/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-token\") pod \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") "
Oct 31 01:53:51.788846 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.788692    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-workload-certs\") pod \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") "
Oct 31 01:53:51.788846 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.788769    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-kube-api-access-7hdz5\" (UniqueName: \"kubernetes.io/projected/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-kube-api-access-7hdz5\") pod \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") "
Oct 31 01:53:51.788846 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.788805    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-podinfo\") pod \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") "
Oct 31 01:53:51.788846 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.788841    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-envoy\") pod \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") "
Oct 31 01:53:51.789586 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.789005    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-workload-socket\") pod \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") "
Oct 31 01:53:51.789586 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.789047    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-workload-certs" pod "b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f" (UID: "b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:53:51.789586 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.789058    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-credential-socket\") pod \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") "
Oct 31 01:53:51.789586 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.789146    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istiod-ca-cert\") pod \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") "
Oct 31 01:53:51.789586 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.789258    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-credential-socket" pod "b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f" (UID: "b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:53:51.789586 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.789295    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-workload-socket" pod "b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f" (UID: "b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:53:51.789586 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.789366    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-data\") pod \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\" (UID: \"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f\") "
Oct 31 01:53:51.791518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.789711    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istiod-ca-cert" pod "b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f" (UID: "b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:51.791518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.789833    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-data" pod "b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f" (UID: "b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:53:51.791518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.790252    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:51.791518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.790277    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:51.791518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.790291    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:51.791518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.790305    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:51.791518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.790320    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:51.792572 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.792547    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-podinfo" pod "b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f" (UID: "b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:53:51.793414 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.793387    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-kube-api-access-7hdz5" pod "b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f" (UID: "b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f"). InnerVolumeSpecName "kube-api-access-7hdz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:51.793584 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.793470    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-envoy" pod "b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f" (UID: "b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:53:51.793754 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.793735    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-token" pod "b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f" (UID: "b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:51.891334 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.891298    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:51.891334 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.891331    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hdz5\" (UniqueName: \"kubernetes.io/projected/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-kube-api-access-7hdz5\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:51.891334 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.891342    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:51.891596 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:51.891351    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:52.658002 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:52.657972    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"
Oct 31 01:53:52.658002 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:52.657982    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js" event={"ID":"b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f","Type":"ContainerDied","Data":"122430080a2395fcf27f39050fc64c6d95ef8a8c929ba804060c62ffb8c23960"}
Oct 31 01:53:52.658534 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:52.658043    5650 scope.go:122] "RemoveContainer" containerID="5c7214bc981078f01dbad8bf089d15f5625b711aa105ae8916982c91e212b464"
Oct 31 01:53:52.684714 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:52.684683    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"]
Oct 31 01:53:52.692757 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:52.692731    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6k8js"]
Oct 31 01:53:54.120061 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.120025    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f" path="/var/lib/kubelet/pods/b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f/volumes"
Oct 31 01:53:54.631960 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.631927    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"]
Oct 31 01:53:54.632612 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.632595    5650 cpu_manager.go:414] "containerMap: removing container" podUID="b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f" containerName="istio-proxy"
Oct 31 01:53:54.632684 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.632613    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f" containerName="istio-proxy"
Oct 31 01:53:54.632684 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.632629    5650 cpu_manager.go:414] "containerMap: removing container" podUID="21539936-858a-45d4-8dcf-c4e15813af0c" containerName="revoke-keys"
Oct 31 01:53:54.632684 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.632635    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21539936-858a-45d4-8dcf-c4e15813af0c" containerName="revoke-keys"
Oct 31 01:53:54.632789 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.632729    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="21539936-858a-45d4-8dcf-c4e15813af0c" containerName="revoke-keys"
Oct 31 01:53:54.632789 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.632748    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8cc67d3-dcd9-4f63-94b1-b45df6a30e3f" containerName="istio-proxy"
Oct 31 01:53:54.638115 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.638086    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.654747 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.654713    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"]
Oct 31 01:53:54.736242 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.736200    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.736242 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.736246    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.736534 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.736402    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.736534 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.736483    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.736665 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.736596    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ktrt\" (UniqueName: \"kubernetes.io/projected/c7b45430-8687-4cbb-b68c-f67723dfa9f6-kube-api-access-4ktrt\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.736665 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.736636    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.736773 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.736758    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.736816 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.736800    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.736900 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.736885    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.838097 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.838013    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.838097 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.838070    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.838097 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.838101    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.838399 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.838190    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.838399 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.838214    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.838399 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.838316    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.838399 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.838346    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.838399 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.838392    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ktrt\" (UniqueName: \"kubernetes.io/projected/c7b45430-8687-4cbb-b68c-f67723dfa9f6-kube-api-access-4ktrt\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.838633 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.838420    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.838633 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.838570    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.838839 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.838808    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.838973 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.838901    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.839040 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.838976    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.839156 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.839135    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.840619 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.840600    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.841383 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.841358    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.852544 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.852518    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.852651 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.852601    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ktrt\" (UniqueName: \"kubernetes.io/projected/c7b45430-8687-4cbb-b68c-f67723dfa9f6-kube-api-access-4ktrt\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:54.963565 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:54.960746    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:55.375396 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:55.375360    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"]
Oct 31 01:53:55.375833 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:53:55.375626    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b45430_8687_4cbb_b68c_f67723dfa9f6.slice/crio-861de6decb9994a7ac3e1501db2cdc119ed22572dd40693c13a05748a17f054d WatchSource:0}: Error finding container 861de6decb9994a7ac3e1501db2cdc119ed22572dd40693c13a05748a17f054d: Status 404 returned error can't find the container with id 861de6decb9994a7ac3e1501db2cdc119ed22572dd40693c13a05748a17f054d
Oct 31 01:53:55.378745 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:55.378705    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:53:55.378917 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:55.378793    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:53:55.378917 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:55.378836    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:53:55.694365 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:55.694076    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5"]
Oct 31 01:53:55.694718 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:55.694605    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5" podUID="be03a565-244e-4052-a13c-97bf29318774" containerName="payload-processing" containerID="cri-o://989c07952ffe3bdc31a0ac1e409337cb942db5b6f9793c873e2f93ba60de7fe4" gracePeriod=30
Oct 31 01:53:55.699850 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:55.699823    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd"]
Oct 31 01:53:55.700161 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:55.700133    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd" podUID="7bd7bede-09f7-45a6-9615-d98483151c9f" containerName="payload-pre-processing" containerID="cri-o://62c5e76358dfadfdf118f6447974d78b60f83c4f26168c055b54fbc691b1a9dc" gracePeriod=30
Oct 31 01:53:55.704311 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:55.704280    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6" event={"ID":"c7b45430-8687-4cbb-b68c-f67723dfa9f6","Type":"ContainerStarted","Data":"db76364024cfe8bf5c03700d35c05029cbd2dc9a1b8b15107ec5cfb4bbaf8ec1"}
Oct 31 01:53:55.704505 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:55.704343    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6" event={"ID":"c7b45430-8687-4cbb-b68c-f67723dfa9f6","Type":"ContainerStarted","Data":"861de6decb9994a7ac3e1501db2cdc119ed22572dd40693c13a05748a17f054d"}
Oct 31 01:53:55.785711 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:55.785649    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6" podStartSLOduration=1.785631583 podStartE2EDuration="1.785631583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:55.76004826 +0000 UTC m=+1991.763762553" watchObservedRunningTime="2026-10-31 01:53:55.785631583 +0000 UTC m=+1991.789345876"
Oct 31 01:53:55.980357 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:55.964604    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:53:56.179395 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:56.157911    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5"
Oct 31 01:53:56.401487 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:56.362001    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be03a565-244e-4052-a13c-97bf29318774-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be03a565-244e-4052-a13c-97bf29318774-plugins-config-volume\") pod \"be03a565-244e-4052-a13c-97bf29318774\" (UID: \"be03a565-244e-4052-a13c-97bf29318774\") "
Oct 31 01:53:56.401487 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:56.362243    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be03a565-244e-4052-a13c-97bf29318774-kube-api-access-44xjv\" (UniqueName: \"kubernetes.io/projected/be03a565-244e-4052-a13c-97bf29318774-kube-api-access-44xjv\") pod \"be03a565-244e-4052-a13c-97bf29318774\" (UID: \"be03a565-244e-4052-a13c-97bf29318774\") "
Oct 31 01:53:56.401487 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:56.378885    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be03a565-244e-4052-a13c-97bf29318774-plugins-config-volume" pod "be03a565-244e-4052-a13c-97bf29318774" (UID: "be03a565-244e-4052-a13c-97bf29318774"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:56.502513 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:56.444656    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be03a565-244e-4052-a13c-97bf29318774-kube-api-access-44xjv" pod "be03a565-244e-4052-a13c-97bf29318774" (UID: "be03a565-244e-4052-a13c-97bf29318774"). InnerVolumeSpecName "kube-api-access-44xjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:56.502513 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:56.468922    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44xjv\" (UniqueName: \"kubernetes.io/projected/be03a565-244e-4052-a13c-97bf29318774-kube-api-access-44xjv\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:56.502513 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:56.468965    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be03a565-244e-4052-a13c-97bf29318774-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:56.832462 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:56.832335    5650 generic.go:356] "Generic (PLEG): container finished" podID="be03a565-244e-4052-a13c-97bf29318774" containerID="989c07952ffe3bdc31a0ac1e409337cb942db5b6f9793c873e2f93ba60de7fe4" exitCode=0
Oct 31 01:53:56.832863 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:56.832809    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5" event={"ID":"be03a565-244e-4052-a13c-97bf29318774","Type":"ContainerDied","Data":"989c07952ffe3bdc31a0ac1e409337cb942db5b6f9793c873e2f93ba60de7fe4"}
Oct 31 01:53:56.833071 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:56.833032    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5" event={"ID":"be03a565-244e-4052-a13c-97bf29318774","Type":"ContainerDied","Data":"f0376b21ec25167f3ca71509ef4e158a613b9691a72511489c42e0a6a951cbe0"}
Oct 31 01:53:56.833257 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:56.833214    5650 scope.go:122] "RemoveContainer" containerID="989c07952ffe3bdc31a0ac1e409337cb942db5b6f9793c873e2f93ba60de7fe4"
Oct 31 01:53:56.833625 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:56.833578    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5"
Oct 31 01:53:56.845509 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:56.841714    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd"
Oct 31 01:53:57.142199 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.142086    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:57.205605 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.151104    5650 generic.go:356] "Generic (PLEG): container finished" podID="7bd7bede-09f7-45a6-9615-d98483151c9f" containerID="62c5e76358dfadfdf118f6447974d78b60f83c4f26168c055b54fbc691b1a9dc" exitCode=0
Oct 31 01:53:57.279489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.151138    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd" event={"ID":"7bd7bede-09f7-45a6-9615-d98483151c9f","Type":"ContainerDied","Data":"62c5e76358dfadfdf118f6447974d78b60f83c4f26168c055b54fbc691b1a9dc"}
Oct 31 01:53:57.279489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.275112    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd" event={"ID":"7bd7bede-09f7-45a6-9615-d98483151c9f","Type":"ContainerDied","Data":"9765d67b0731c142a749ded551c168f66748302f2890226c23818bd0a6c54ce8"}
Oct 31 01:53:57.279489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.156474    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd7bede-09f7-45a6-9615-d98483151c9f-kube-api-access-qvl47\" (UniqueName: \"kubernetes.io/projected/7bd7bede-09f7-45a6-9615-d98483151c9f-kube-api-access-qvl47\") pod \"7bd7bede-09f7-45a6-9615-d98483151c9f\" (UID: \"7bd7bede-09f7-45a6-9615-d98483151c9f\") "
Oct 31 01:53:57.279489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.275718    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bd7bede-09f7-45a6-9615-d98483151c9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd7bede-09f7-45a6-9615-d98483151c9f-plugins-config-volume\") pod \"7bd7bede-09f7-45a6-9615-d98483151c9f\" (UID: \"7bd7bede-09f7-45a6-9615-d98483151c9f\") "
Oct 31 01:53:57.279489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.278225    5650 scope.go:122] "RemoveContainer" containerID="989c07952ffe3bdc31a0ac1e409337cb942db5b6f9793c873e2f93ba60de7fe4"
Oct 31 01:53:57.325235 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.280596    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 31 01:53:57.325235 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.280740    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6" podUID="c7b45430-8687-4cbb-b68c-f67723dfa9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 31 01:53:57.378089 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.378041    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd7bede-09f7-45a6-9615-d98483151c9f-plugins-config-volume" pod "7bd7bede-09f7-45a6-9615-d98483151c9f" (UID: "7bd7bede-09f7-45a6-9615-d98483151c9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:57.491992 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.491855    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:57.509488 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.506307    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 01:53:57.555467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.537599    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"]
Oct 31 01:53:57.578472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.560976    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:57.578472 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:53:57.561598    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"989c07952ffe3bdc31a0ac1e409337cb942db5b6f9793c873e2f93ba60de7fe4\": container with ID starting with 989c07952ffe3bdc31a0ac1e409337cb942db5b6f9793c873e2f93ba60de7fe4 not found: ID does not exist" containerID="989c07952ffe3bdc31a0ac1e409337cb942db5b6f9793c873e2f93ba60de7fe4"
Oct 31 01:53:57.578472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.561874    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"989c07952ffe3bdc31a0ac1e409337cb942db5b6f9793c873e2f93ba60de7fe4"} err="failed to get container status \"989c07952ffe3bdc31a0ac1e409337cb942db5b6f9793c873e2f93ba60de7fe4\": rpc error: code = NotFound desc = could not find container \"989c07952ffe3bdc31a0ac1e409337cb942db5b6f9793c873e2f93ba60de7fe4\": container with ID starting with 989c07952ffe3bdc31a0ac1e409337cb942db5b6f9793c873e2f93ba60de7fe4 not found: ID does not exist"
Oct 31 01:53:57.578472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.561906    5650 scope.go:122] "RemoveContainer" containerID="62c5e76358dfadfdf118f6447974d78b60f83c4f26168c055b54fbc691b1a9dc"
Oct 31 01:53:57.609182 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.553933    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:57.622556 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.622512    5650 cpu_manager.go:414] "containerMap: removing container" podUID="be03a565-244e-4052-a13c-97bf29318774" containerName="payload-processing"
Oct 31 01:53:57.663471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.643769    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be03a565-244e-4052-a13c-97bf29318774" containerName="payload-processing"
Oct 31 01:53:57.663471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.643876    5650 cpu_manager.go:414] "containerMap: removing container" podUID="7bd7bede-09f7-45a6-9615-d98483151c9f" containerName="payload-pre-processing"
Oct 31 01:53:57.663471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.643886    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd7bede-09f7-45a6-9615-d98483151c9f" containerName="payload-pre-processing"
Oct 31 01:53:57.663471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.644361    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd7bede-09f7-45a6-9615-d98483151c9f" containerName="payload-pre-processing"
Oct 31 01:53:57.663471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.644423    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="be03a565-244e-4052-a13c-97bf29318774" containerName="payload-processing"
Oct 31 01:53:57.663471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.659887    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd7bede-09f7-45a6-9615-d98483151c9f-kube-api-access-qvl47" pod "7bd7bede-09f7-45a6-9615-d98483151c9f" (UID: "7bd7bede-09f7-45a6-9615-d98483151c9f"). InnerVolumeSpecName "kube-api-access-qvl47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:57.664483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.627849    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd7bede-09f7-45a6-9615-d98483151c9f-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:57.776062 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.776018    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:57.799382 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.799342    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvl47\" (UniqueName: \"kubernetes.io/projected/7bd7bede-09f7-45a6-9615-d98483151c9f-kube-api-access-qvl47\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:53:57.840127 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.840080    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"]
Oct 31 01:53:57.840546 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:57.840511    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.052528 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.052410    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwkpj\" (UniqueName: \"kubernetes.io/projected/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-kube-api-access-cwkpj\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.052961 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.052930    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-podinfo\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.053198 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.053178    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-credential-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.053483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.053460    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-data\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.053705 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.053684    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-workload-certs\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.057468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.053881    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istiod-ca-cert\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.057783 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.057760    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-workload-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.061530 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.061505    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-token\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.061751 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.061733    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-envoy\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.101472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.088486    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 31 01:53:58.101836 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.101792    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6" podUID="c7b45430-8687-4cbb-b68c-f67723dfa9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 31 01:53:58.113490 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.113392    5650 scope.go:122] "RemoveContainer" containerID="62c5e76358dfadfdf118f6447974d78b60f83c4f26168c055b54fbc691b1a9dc"
Oct 31 01:53:58.114543 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:53:58.114507    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62c5e76358dfadfdf118f6447974d78b60f83c4f26168c055b54fbc691b1a9dc\": container with ID starting with 62c5e76358dfadfdf118f6447974d78b60f83c4f26168c055b54fbc691b1a9dc not found: ID does not exist" containerID="62c5e76358dfadfdf118f6447974d78b60f83c4f26168c055b54fbc691b1a9dc"
Oct 31 01:53:58.114757 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.114728    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62c5e76358dfadfdf118f6447974d78b60f83c4f26168c055b54fbc691b1a9dc"} err="failed to get container status \"62c5e76358dfadfdf118f6447974d78b60f83c4f26168c055b54fbc691b1a9dc\": rpc error: code = NotFound desc = could not find container \"62c5e76358dfadfdf118f6447974d78b60f83c4f26168c055b54fbc691b1a9dc\": container with ID starting with 62c5e76358dfadfdf118f6447974d78b60f83c4f26168c055b54fbc691b1a9dc not found: ID does not exist"
Oct 31 01:53:58.151826 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.151775    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:58.153058 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.152098    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:58.153842 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.153813    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:58.159576 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.152128    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 01:53:58.161409 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.161375    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 01:53:58.169144 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.169115    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-credential-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.169507 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.169482    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-data\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.169742 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.169725    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-workload-certs\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.169927 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.169894    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istiod-ca-cert\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.170136 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.170119    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-workload-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.170318 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.170288    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-token\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.170532 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.170504    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-envoy\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.189939 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.171552    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-credential-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.190177 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.171867    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-data\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.190323 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.172157    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-workload-certs\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.190440 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.177032    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istiod-ca-cert\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.190576 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.187542    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-workload-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.190698 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.190518    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwkpj\" (UniqueName: \"kubernetes.io/projected/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-kube-api-access-cwkpj\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.214471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.190987    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-podinfo\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.214856 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.214039    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-podinfo\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.238489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.221352    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-envoy\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.312523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.303262    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwkpj\" (UniqueName: \"kubernetes.io/projected/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-kube-api-access-cwkpj\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.312772 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.309073    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-token\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.387474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.353584    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:53:58.387474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.380148    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5"]
Oct 31 01:53:58.392249 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.392175    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:53:58.409659 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.409613    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd"
Oct 31 01:53:58.508469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.507578    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:58.520833 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.520786    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:53:58.601524 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.555639    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:58.644477 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.565155    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:58.663907 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.663860    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:53:58.664179 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.615314    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:58.664827 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.664383    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:53:58.664827 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.664428    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-925bac-5c47f5688d-qwwc5"]
Oct 31 01:53:58.687667 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.686823    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:53:58.865912 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.744815    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:58.981841 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.807634    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:58.987607 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.987575    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:53:59.042474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:58.995571    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 31 01:53:59.066472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:59.042850    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6" podUID="c7b45430-8687-4cbb-b68c-f67723dfa9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 31 01:53:59.222339 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:59.185773    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:59.264175 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:59.223156    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:59.264175 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:59.223487    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 01:53:59.264175 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:59.224004    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:53:59.264175 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:59.224066    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:53:59.266482 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:59.266425    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 01:53:59.276812 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:59.267091    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:53:59.349082 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:53:59.325750    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:53:59.621471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:59.558510    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:59.871533 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:59.656580    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:59.871533 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:59.656617    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 01:53:59.873190 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:59.665278    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:00.029975 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:59.983804    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 31 01:54:00.029975 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:53:59.983863    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6" podUID="c7b45430-8687-4cbb-b68c-f67723dfa9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 31 01:54:00.029975 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:00.013041    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:54:00.029975 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:00.013105    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:54:00.067700 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:00.031074    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 31 01:54:00.290547 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:00.290012    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:00.361808 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:54:00.349366    5650 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe03a565_244e_4052_a13c_97bf29318774.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe03a565_244e_4052_a13c_97bf29318774.slice/crio-f0376b21ec25167f3ca71509ef4e158a613b9691a72511489c42e0a6a951cbe0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd7bede_09f7_45a6_9615_d98483151c9f.slice/crio-9765d67b0731c142a749ded551c168f66748302f2890226c23818bd0a6c54ce8\": RecentStats: unable to find data in memory cache]"
Oct 31 01:54:00.486337 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:00.486292    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be03a565-244e-4052-a13c-97bf29318774" path="/var/lib/kubelet/pods/be03a565-244e-4052-a13c-97bf29318774/volumes"
Oct 31 01:54:00.486920 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:00.486892    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:54:00.487055 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:00.486978    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:54:00.487055 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:00.487045    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:54:00.487331 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:00.487093    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:54:00.487331 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:00.487140    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:54:00.487331 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:00.487184    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:54:00.496767 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:00.496727    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:54:00.496911 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:00.496836    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:54:00.496911 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:00.496893    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:54:00.523795 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:00.523654    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:54:00.582533 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:00.582502    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd"]
Oct 31 01:54:00.591417 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:00.591355    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-925bac-76cc94458c-lffmd"]
Oct 31 01:54:00.663605 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:00.663579    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"]
Oct 31 01:54:00.682769 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:00.682727    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj" event={"ID":"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2","Type":"ContainerStarted","Data":"03f83f577d9adbf2dffc7bac4ebd0e77bd9171909245b74d40cd6d90247f3355"}
Oct 31 01:54:00.683017 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:00.682801    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj" event={"ID":"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2","Type":"ContainerStarted","Data":"9c776b58998f2eac1401a3010dbb4fe94625d8631da3562755862b0ccbfc5443"}
Oct 31 01:54:00.792638 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:00.792550    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj" podStartSLOduration=4.792518211 podStartE2EDuration="4.792518211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:00.788132492 +0000 UTC m=+1996.791846782" watchObservedRunningTime="2026-10-31 01:54:00.792518211 +0000 UTC m=+1996.796232504"
Oct 31 01:54:00.962809 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:00.962776    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 31 01:54:00.962978 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:00.962835    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6" podUID="c7b45430-8687-4cbb-b68c-f67723dfa9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 31 01:54:01.394080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:01.393993    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:54:01.400763 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:01.400732    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:54:01.694441 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:01.694354    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:54:01.695557 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:01.695535    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:54:01.961868 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:01.961827    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 31 01:54:01.962033 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:01.961890    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6" podUID="c7b45430-8687-4cbb-b68c-f67723dfa9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 31 01:54:02.132859 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:02.132824    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd7bede-09f7-45a6-9615-d98483151c9f" path="/var/lib/kubelet/pods/7bd7bede-09f7-45a6-9615-d98483151c9f/volumes"
Oct 31 01:54:02.961863 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:02.961824    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 31 01:54:02.962385 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:02.961890    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6" podUID="c7b45430-8687-4cbb-b68c-f67723dfa9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 31 01:54:04.195803 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:04.195762    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk"]
Oct 31 01:54:04.201660 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:04.201631    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk"
Oct 31 01:54:04.228049 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:04.228010    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:54:04.228229 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:04.228088    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:54:04.234289 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:04.234259    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk"]
Oct 31 01:54:04.341299 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:04.341267    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b281c0e5-f39a-4a24-ae87-f6c346803b52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk\" (UID: \"b281c0e5-f39a-4a24-ae87-f6c346803b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk"
Oct 31 01:54:04.341523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:04.341403    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fhjb\" (UniqueName: \"kubernetes.io/projected/b281c0e5-f39a-4a24-ae87-f6c346803b52-kube-api-access-6fhjb\") pod \"maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk\" (UID: \"b281c0e5-f39a-4a24-ae87-f6c346803b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk"
Oct 31 01:54:04.341577 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:04.341546    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b281c0e5-f39a-4a24-ae87-f6c346803b52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk\" (UID: \"b281c0e5-f39a-4a24-ae87-f6c346803b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk"
Oct 31 01:54:04.401309 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:04.401226    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"]
Oct 31 01:54:04.401540 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:04.401432    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f" podUID="4519a56f-4611-4338-aa8c-aad734828add" containerName="istio-proxy" containerID="cri-o://8b596584859f5e5149231fde325164683182ea8e3ddd44d45806bfbdffd1d16a" gracePeriod=30
Oct 31 01:54:04.443402 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:04.443369    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fhjb\" (UniqueName: \"kubernetes.io/projected/b281c0e5-f39a-4a24-ae87-f6c346803b52-kube-api-access-6fhjb\") pod \"maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk\" (UID: \"b281c0e5-f39a-4a24-ae87-f6c346803b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk"
Oct 31 01:54:04.443608 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:04.443480    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b281c0e5-f39a-4a24-ae87-f6c346803b52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk\" (UID: \"b281c0e5-f39a-4a24-ae87-f6c346803b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk"
Oct 31 01:54:04.443660 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:04.443632    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b281c0e5-f39a-4a24-ae87-f6c346803b52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk\" (UID: \"b281c0e5-f39a-4a24-ae87-f6c346803b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk"
Oct 31 01:54:04.444184 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:04.444156    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b281c0e5-f39a-4a24-ae87-f6c346803b52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk\" (UID: \"b281c0e5-f39a-4a24-ae87-f6c346803b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk"
Oct 31 01:54:04.446112 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:04.446053    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b281c0e5-f39a-4a24-ae87-f6c346803b52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk\" (UID: \"b281c0e5-f39a-4a24-ae87-f6c346803b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk"
Oct 31 01:54:04.459396 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:04.459295    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fhjb\" (UniqueName: \"kubernetes.io/projected/b281c0e5-f39a-4a24-ae87-f6c346803b52-kube-api-access-6fhjb\") pod \"maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk\" (UID: \"b281c0e5-f39a-4a24-ae87-f6c346803b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk"
Oct 31 01:54:04.523834 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:04.523787    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk"
Oct 31 01:54:04.867386 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:04.867341    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk"]
Oct 31 01:54:05.004673 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:05.004630    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:54:05.004874 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:05.004713    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6" podUID="c7b45430-8687-4cbb-b68c-f67723dfa9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:05.468397 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:05.468363    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:54:05.468917 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:05.468775    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:54:05.469774 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:05.469748    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 01:54:05.733267 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:05.733225    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk" event={"ID":"b281c0e5-f39a-4a24-ae87-f6c346803b52","Type":"ContainerStarted","Data":"07cc2ab40944aadf5b143ff44eb044cd0569d2851ee3591b04dd41190f349d4d"}
Oct 31 01:54:05.733437 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:05.733295    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk" event={"ID":"b281c0e5-f39a-4a24-ae87-f6c346803b52","Type":"ContainerStarted","Data":"030ac93084e9e9b1d4304e8085d86614b97381691fa18dce38817a2e7728ffc0"}
Oct 31 01:54:05.770656 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:05.770594    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk" podStartSLOduration=1.77058045 podStartE2EDuration="1.77058045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:05.76533111 +0000 UTC m=+2001.769045408" watchObservedRunningTime="2026-10-31 01:54:05.77058045 +0000 UTC m=+2001.774294743"
Oct 31 01:54:06.526704 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.526666    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"]
Oct 31 01:54:06.531081 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.531055    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.542408 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.542382    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-eb8ae6-openshift-default-dockercfg-h5vlp\""
Oct 31 01:54:06.548684 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.548656    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"]
Oct 31 01:54:06.720499 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.712701    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-token\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.720499 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.712779    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd7k8\" (UniqueName: \"kubernetes.io/projected/7fe243de-3376-4d8d-87bd-032e29f0083d-kube-api-access-xd7k8\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.720499 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.712882    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-credential-socket\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.720499 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.712930    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-podinfo\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.720499 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.713207    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-data\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.720499 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.713341    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-workload-socket\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.720499 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.713433    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-workload-certs\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.720499 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.713511    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-envoy\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.720499 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.713539    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe243de-3376-4d8d-87bd-032e29f0083d-istiod-ca-cert\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.770471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.765591    5650 generic.go:356] "Generic (PLEG): container finished" podID="b281c0e5-f39a-4a24-ae87-f6c346803b52" containerID="07cc2ab40944aadf5b143ff44eb044cd0569d2851ee3591b04dd41190f349d4d" exitCode=0
Oct 31 01:54:06.770471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.765786    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk" event={"ID":"b281c0e5-f39a-4a24-ae87-f6c346803b52","Type":"ContainerDied","Data":"07cc2ab40944aadf5b143ff44eb044cd0569d2851ee3591b04dd41190f349d4d"}
Oct 31 01:54:06.819617 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.819529    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-workload-certs\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.819617 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.819587    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-envoy\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.819617 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.819611    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe243de-3376-4d8d-87bd-032e29f0083d-istiod-ca-cert\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.819851 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.819725    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-token\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.819851 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.819746    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd7k8\" (UniqueName: \"kubernetes.io/projected/7fe243de-3376-4d8d-87bd-032e29f0083d-kube-api-access-xd7k8\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.819851 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.819778    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-credential-socket\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.819851 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.819802    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-podinfo\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.819980 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.819918    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-data\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.819980 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.819955    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-workload-socket\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.820070 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.820018    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-workload-certs\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.820431 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.820376    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-credential-socket\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.820587 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.820439    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-data\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.820793 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.820770    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-workload-socket\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.821088 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.821063    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe243de-3376-4d8d-87bd-032e29f0083d-istiod-ca-cert\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.825389 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.825364    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-envoy\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.831717 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.831690    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-podinfo\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.837512 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.836782    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd7k8\" (UniqueName: \"kubernetes.io/projected/7fe243de-3376-4d8d-87bd-032e29f0083d-kube-api-access-xd7k8\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.837512 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.836890    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-token\") pod \"e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") " pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:06.847510 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:06.847437    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:07.178511 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:07.164114    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"]
Oct 31 01:54:07.181508 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:07.179630    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:54:07.181508 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:07.179729    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:54:07.181508 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:07.179772    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:54:07.776638 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:07.776599    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w" event={"ID":"7fe243de-3376-4d8d-87bd-032e29f0083d","Type":"ContainerStarted","Data":"f7de8e8e314a30de871cf03eccd5bf5a7fd6b496e395ebc93be647095e74431a"}
Oct 31 01:54:07.777234 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:07.776662    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w" event={"ID":"7fe243de-3376-4d8d-87bd-032e29f0083d","Type":"ContainerStarted","Data":"1d12b9dff0656d1ccdda95e80804ba35e03d4a54b4c3f79837f3941dad47c3bc"}
Oct 31 01:54:07.819970 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:07.819899    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w" podStartSLOduration=1.819877631 podStartE2EDuration="1.819877631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:07.816952657 +0000 UTC m=+2003.820666940" watchObservedRunningTime="2026-10-31 01:54:07.819877631 +0000 UTC m=+2003.823591923"
Oct 31 01:54:07.850365 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:07.849553    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:07.952201 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:07.952170    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk"
Oct 31 01:54:08.053426 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:08.053334    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b281c0e5-f39a-4a24-ae87-f6c346803b52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b281c0e5-f39a-4a24-ae87-f6c346803b52-maas-api-openshift-ca\") pod \"b281c0e5-f39a-4a24-ae87-f6c346803b52\" (UID: \"b281c0e5-f39a-4a24-ae87-f6c346803b52\") "
Oct 31 01:54:08.053627 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:08.053608    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b281c0e5-f39a-4a24-ae87-f6c346803b52-kube-api-access-6fhjb\" (UniqueName: \"kubernetes.io/projected/b281c0e5-f39a-4a24-ae87-f6c346803b52-kube-api-access-6fhjb\") pod \"b281c0e5-f39a-4a24-ae87-f6c346803b52\" (UID: \"b281c0e5-f39a-4a24-ae87-f6c346803b52\") "
Oct 31 01:54:08.053702 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:08.053676    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b281c0e5-f39a-4a24-ae87-f6c346803b52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b281c0e5-f39a-4a24-ae87-f6c346803b52-maas-api-xks-ca\") pod \"b281c0e5-f39a-4a24-ae87-f6c346803b52\" (UID: \"b281c0e5-f39a-4a24-ae87-f6c346803b52\") "
Oct 31 01:54:08.053935 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:08.053911    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b281c0e5-f39a-4a24-ae87-f6c346803b52-maas-api-openshift-ca" pod "b281c0e5-f39a-4a24-ae87-f6c346803b52" (UID: "b281c0e5-f39a-4a24-ae87-f6c346803b52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:08.054473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:08.054431    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b281c0e5-f39a-4a24-ae87-f6c346803b52-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:08.056537 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:08.056499    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b281c0e5-f39a-4a24-ae87-f6c346803b52-kube-api-access-6fhjb" pod "b281c0e5-f39a-4a24-ae87-f6c346803b52" (UID: "b281c0e5-f39a-4a24-ae87-f6c346803b52"). InnerVolumeSpecName "kube-api-access-6fhjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:08.056664 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:08.056648    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b281c0e5-f39a-4a24-ae87-f6c346803b52-maas-api-xks-ca" pod "b281c0e5-f39a-4a24-ae87-f6c346803b52" (UID: "b281c0e5-f39a-4a24-ae87-f6c346803b52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:54:08.156473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:08.156383    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fhjb\" (UniqueName: \"kubernetes.io/projected/b281c0e5-f39a-4a24-ae87-f6c346803b52-kube-api-access-6fhjb\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:08.156473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:08.156475    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b281c0e5-f39a-4a24-ae87-f6c346803b52-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:08.786806 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:08.786776    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk"
Oct 31 01:54:08.787408 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:08.786848    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk" event={"ID":"b281c0e5-f39a-4a24-ae87-f6c346803b52","Type":"ContainerDied","Data":"030ac93084e9e9b1d4304e8085d86614b97381691fa18dce38817a2e7728ffc0"}
Oct 31 01:54:08.787408 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:08.786952    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="030ac93084e9e9b1d4304e8085d86614b97381691fa18dce38817a2e7728ffc0"
Oct 31 01:54:08.854922 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:08.854890    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:09.709683 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.709647    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:54:09.780806 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.780774    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-istio-data\") pod \"4519a56f-4611-4338-aa8c-aad734828add\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") "
Oct 31 01:54:09.780999 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.780880    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4519a56f-4611-4338-aa8c-aad734828add-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4519a56f-4611-4338-aa8c-aad734828add-istio-podinfo\") pod \"4519a56f-4611-4338-aa8c-aad734828add\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") "
Oct 31 01:54:09.780999 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.780982    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4519a56f-4611-4338-aa8c-aad734828add-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4519a56f-4611-4338-aa8c-aad734828add-istiod-ca-cert\") pod \"4519a56f-4611-4338-aa8c-aad734828add\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") "
Oct 31 01:54:09.781512 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.781128    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-istio-data" pod "4519a56f-4611-4338-aa8c-aad734828add" (UID: "4519a56f-4611-4338-aa8c-aad734828add"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:09.781512 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.781162    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-istio-envoy\") pod \"4519a56f-4611-4338-aa8c-aad734828add\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") "
Oct 31 01:54:09.781512 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.781210    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-workload-certs\") pod \"4519a56f-4611-4338-aa8c-aad734828add\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") "
Oct 31 01:54:09.781512 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.781428    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4519a56f-4611-4338-aa8c-aad734828add-istiod-ca-cert" pod "4519a56f-4611-4338-aa8c-aad734828add" (UID: "4519a56f-4611-4338-aa8c-aad734828add"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:09.781512 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.781480    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-workload-certs" pod "4519a56f-4611-4338-aa8c-aad734828add" (UID: "4519a56f-4611-4338-aa8c-aad734828add"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:09.781512 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.781247    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-workload-socket\") pod \"4519a56f-4611-4338-aa8c-aad734828add\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") "
Oct 31 01:54:09.781993 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.781629    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4519a56f-4611-4338-aa8c-aad734828add-kube-api-access-shqp8\" (UniqueName: \"kubernetes.io/projected/4519a56f-4611-4338-aa8c-aad734828add-kube-api-access-shqp8\") pod \"4519a56f-4611-4338-aa8c-aad734828add\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") "
Oct 31 01:54:09.782061 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.781990    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4519a56f-4611-4338-aa8c-aad734828add-istio-token\" (UniqueName: \"kubernetes.io/projected/4519a56f-4611-4338-aa8c-aad734828add-istio-token\") pod \"4519a56f-4611-4338-aa8c-aad734828add\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") "
Oct 31 01:54:09.782477 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.782425    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-credential-socket\") pod \"4519a56f-4611-4338-aa8c-aad734828add\" (UID: \"4519a56f-4611-4338-aa8c-aad734828add\") "
Oct 31 01:54:09.783025 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.783007    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:09.783127 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.783029    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4519a56f-4611-4338-aa8c-aad734828add-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:09.783127 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.783044    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:09.783275 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.781666    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-workload-socket" pod "4519a56f-4611-4338-aa8c-aad734828add" (UID: "4519a56f-4611-4338-aa8c-aad734828add"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:09.783337 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.783318    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-credential-socket" pod "4519a56f-4611-4338-aa8c-aad734828add" (UID: "4519a56f-4611-4338-aa8c-aad734828add"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:09.783810 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.783785    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-istio-envoy" pod "4519a56f-4611-4338-aa8c-aad734828add" (UID: "4519a56f-4611-4338-aa8c-aad734828add"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:09.785662 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.785553    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4519a56f-4611-4338-aa8c-aad734828add-istio-podinfo" pod "4519a56f-4611-4338-aa8c-aad734828add" (UID: "4519a56f-4611-4338-aa8c-aad734828add"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:54:09.785662 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.785617    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4519a56f-4611-4338-aa8c-aad734828add-istio-token" pod "4519a56f-4611-4338-aa8c-aad734828add" (UID: "4519a56f-4611-4338-aa8c-aad734828add"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:09.785662 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.785647    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4519a56f-4611-4338-aa8c-aad734828add-kube-api-access-shqp8" pod "4519a56f-4611-4338-aa8c-aad734828add" (UID: "4519a56f-4611-4338-aa8c-aad734828add"). InnerVolumeSpecName "kube-api-access-shqp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:09.799104 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.799047    5650 generic.go:356] "Generic (PLEG): container finished" podID="4519a56f-4611-4338-aa8c-aad734828add" containerID="8b596584859f5e5149231fde325164683182ea8e3ddd44d45806bfbdffd1d16a" exitCode=0
Oct 31 01:54:09.799426 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.799117    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"
Oct 31 01:54:09.799426 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.799128    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f" event={"ID":"4519a56f-4611-4338-aa8c-aad734828add","Type":"ContainerDied","Data":"8b596584859f5e5149231fde325164683182ea8e3ddd44d45806bfbdffd1d16a"}
Oct 31 01:54:09.799426 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.799189    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f" event={"ID":"4519a56f-4611-4338-aa8c-aad734828add","Type":"ContainerDied","Data":"7098b66c5351e1ae0fc91ca206b09c9aa89c001a5c449b9ee268a938787df534"}
Oct 31 01:54:09.799426 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.799217    5650 scope.go:122] "RemoveContainer" containerID="8b596584859f5e5149231fde325164683182ea8e3ddd44d45806bfbdffd1d16a"
Oct 31 01:54:09.799856 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.799840    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:09.801283 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.801145    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:54:09.826710 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.826666    5650 scope.go:122] "RemoveContainer" containerID="8b596584859f5e5149231fde325164683182ea8e3ddd44d45806bfbdffd1d16a"
Oct 31 01:54:09.827086 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:54:09.827061    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b596584859f5e5149231fde325164683182ea8e3ddd44d45806bfbdffd1d16a\": container with ID starting with 8b596584859f5e5149231fde325164683182ea8e3ddd44d45806bfbdffd1d16a not found: ID does not exist" containerID="8b596584859f5e5149231fde325164683182ea8e3ddd44d45806bfbdffd1d16a"
Oct 31 01:54:09.827178 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.827097    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b596584859f5e5149231fde325164683182ea8e3ddd44d45806bfbdffd1d16a"} err="failed to get container status \"8b596584859f5e5149231fde325164683182ea8e3ddd44d45806bfbdffd1d16a\": rpc error: code = NotFound desc = could not find container \"8b596584859f5e5149231fde325164683182ea8e3ddd44d45806bfbdffd1d16a\": container with ID starting with 8b596584859f5e5149231fde325164683182ea8e3ddd44d45806bfbdffd1d16a not found: ID does not exist"
Oct 31 01:54:09.877351 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.864896    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"]
Oct 31 01:54:09.877351 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.877122    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-jcb6f"]
Oct 31 01:54:09.884584 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.884553    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4519a56f-4611-4338-aa8c-aad734828add-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:09.884584 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.884581    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:09.884867 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.884598    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:09.884867 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.884610    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shqp8\" (UniqueName: \"kubernetes.io/projected/4519a56f-4611-4338-aa8c-aad734828add-kube-api-access-shqp8\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:09.884867 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.884626    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4519a56f-4611-4338-aa8c-aad734828add-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:09.884867 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:09.884637    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4519a56f-4611-4338-aa8c-aad734828add-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:10.121928 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:10.121888    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4519a56f-4611-4338-aa8c-aad734828add" path="/var/lib/kubelet/pods/4519a56f-4611-4338-aa8c-aad734828add/volumes"
Oct 31 01:54:11.585665 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.585421    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4"]
Oct 31 01:54:11.586421 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.586396    5650 cpu_manager.go:414] "containerMap: removing container" podUID="4519a56f-4611-4338-aa8c-aad734828add" containerName="istio-proxy"
Oct 31 01:54:11.586421 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.586420    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4519a56f-4611-4338-aa8c-aad734828add" containerName="istio-proxy"
Oct 31 01:54:11.586608 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.586436    5650 cpu_manager.go:414] "containerMap: removing container" podUID="b281c0e5-f39a-4a24-ae87-f6c346803b52" containerName="revoke-keys"
Oct 31 01:54:11.586608 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.586459    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b281c0e5-f39a-4a24-ae87-f6c346803b52" containerName="revoke-keys"
Oct 31 01:54:11.586608 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.586597    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="4519a56f-4611-4338-aa8c-aad734828add" containerName="istio-proxy"
Oct 31 01:54:11.586608 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.586605    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="b281c0e5-f39a-4a24-ae87-f6c346803b52" containerName="revoke-keys"
Oct 31 01:54:11.591192 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.591171    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4"
Oct 31 01:54:11.603407 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.603112    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-eb8ae6\""
Oct 31 01:54:11.603407 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.603185    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-eb8ae6-dockercfg-67kwg\""
Oct 31 01:54:11.606246 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.606216    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4"]
Oct 31 01:54:11.620440 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.620408    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv"]
Oct 31 01:54:11.625950 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.625917    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv"
Oct 31 01:54:11.641439 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.641407    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv"]
Oct 31 01:54:11.735469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.734506    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbnng\" (UniqueName: \"kubernetes.io/projected/fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91-kube-api-access-hbnng\") pod \"payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4\" (UID: \"fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4"
Oct 31 01:54:11.735469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.734587    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlk98\" (UniqueName: \"kubernetes.io/projected/9feea519-15e9-4a7b-bd63-ab6bf4965707-kube-api-access-vlk98\") pod \"payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv\" (UID: \"9feea519-15e9-4a7b-bd63-ab6bf4965707\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv"
Oct 31 01:54:11.735469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.734708    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9feea519-15e9-4a7b-bd63-ab6bf4965707-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv\" (UID: \"9feea519-15e9-4a7b-bd63-ab6bf4965707\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv"
Oct 31 01:54:11.735469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.734835    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4\" (UID: \"fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4"
Oct 31 01:54:11.856572 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.856254    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbnng\" (UniqueName: \"kubernetes.io/projected/fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91-kube-api-access-hbnng\") pod \"payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4\" (UID: \"fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4"
Oct 31 01:54:11.856572 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.856340    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlk98\" (UniqueName: \"kubernetes.io/projected/9feea519-15e9-4a7b-bd63-ab6bf4965707-kube-api-access-vlk98\") pod \"payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv\" (UID: \"9feea519-15e9-4a7b-bd63-ab6bf4965707\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv"
Oct 31 01:54:11.856572 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.856469    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9feea519-15e9-4a7b-bd63-ab6bf4965707-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv\" (UID: \"9feea519-15e9-4a7b-bd63-ab6bf4965707\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv"
Oct 31 01:54:11.856885 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.856595    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4\" (UID: \"fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4"
Oct 31 01:54:11.885489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.882080    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9feea519-15e9-4a7b-bd63-ab6bf4965707-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv\" (UID: \"9feea519-15e9-4a7b-bd63-ab6bf4965707\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv"
Oct 31 01:54:11.885489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.883958    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4\" (UID: \"fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4"
Oct 31 01:54:11.900495 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.898244    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlk98\" (UniqueName: \"kubernetes.io/projected/9feea519-15e9-4a7b-bd63-ab6bf4965707-kube-api-access-vlk98\") pod \"payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv\" (UID: \"9feea519-15e9-4a7b-bd63-ab6bf4965707\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv"
Oct 31 01:54:11.902966 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.902404    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbnng\" (UniqueName: \"kubernetes.io/projected/fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91-kube-api-access-hbnng\") pod \"payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4\" (UID: \"fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4"
Oct 31 01:54:11.915692 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.913640    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4"
Oct 31 01:54:11.943685 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:11.943646    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv"
Oct 31 01:54:12.230155 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:12.230111    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4"]
Oct 31 01:54:12.231981 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:54:12.231951    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5bf02e_6f9a_4e20_bea5_6d372ac8ab91.slice/crio-6c56969a5a8cb2853ac42806f41f7d29491926371a9332ca35c76ca362eca307 WatchSource:0}: Error finding container 6c56969a5a8cb2853ac42806f41f7d29491926371a9332ca35c76ca362eca307: Status 404 returned error can't find the container with id 6c56969a5a8cb2853ac42806f41f7d29491926371a9332ca35c76ca362eca307
Oct 31 01:54:12.248939 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:54:12.248901    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9feea519_15e9_4a7b_bd63_ab6bf4965707.slice/crio-651f359ff16384c74ac72a0fff4c3fd42e18673ffcee49aebd669476e224942d WatchSource:0}: Error finding container 651f359ff16384c74ac72a0fff4c3fd42e18673ffcee49aebd669476e224942d: Status 404 returned error can't find the container with id 651f359ff16384c74ac72a0fff4c3fd42e18673ffcee49aebd669476e224942d
Oct 31 01:54:12.250517 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:12.250493    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv"]
Oct 31 01:54:12.845008 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:12.844972    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv" event={"ID":"9feea519-15e9-4a7b-bd63-ab6bf4965707","Type":"ContainerStarted","Data":"fbf9e78e5de5a28e6dd32eda8ae2d7b7f1402036e98a0532b554fc3d3d9f349b"}
Oct 31 01:54:12.845494 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:12.845025    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv" event={"ID":"9feea519-15e9-4a7b-bd63-ab6bf4965707","Type":"ContainerStarted","Data":"651f359ff16384c74ac72a0fff4c3fd42e18673ffcee49aebd669476e224942d"}
Oct 31 01:54:12.845494 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:12.845062    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv"
Oct 31 01:54:12.846980 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:12.846952    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4" event={"ID":"fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91","Type":"ContainerStarted","Data":"0a1779eed9d1a414ae465c5c00a4021e216bab054fe9b11fe4edb2ac13222483"}
Oct 31 01:54:12.847112 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:12.847003    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4" event={"ID":"fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91","Type":"ContainerStarted","Data":"6c56969a5a8cb2853ac42806f41f7d29491926371a9332ca35c76ca362eca307"}
Oct 31 01:54:12.847112 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:12.847052    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4"
Oct 31 01:54:12.869138 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:12.869084    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv" podStartSLOduration=1.869070309 podStartE2EDuration="1.869070309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:12.864842642 +0000 UTC m=+2008.868556946" watchObservedRunningTime="2026-10-31 01:54:12.869070309 +0000 UTC m=+2008.872784600"
Oct 31 01:54:12.890847 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:12.890775    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4" podStartSLOduration=1.8907540090000001 podStartE2EDuration="1.890754009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:12.885621075 +0000 UTC m=+2008.889335365" watchObservedRunningTime="2026-10-31 01:54:12.890754009 +0000 UTC m=+2008.894468302"
Oct 31 01:54:16.548501 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:16.543813    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk"]
Oct 31 01:54:16.548501 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:16.544517    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f65e1-89jzk"]
Oct 31 01:54:18.120242 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:18.120207    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b281c0e5-f39a-4a24-ae87-f6c346803b52" path="/var/lib/kubelet/pods/b281c0e5-f39a-4a24-ae87-f6c346803b52/volumes"
Oct 31 01:54:20.531062 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:20.531021    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"]
Oct 31 01:54:20.531550 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:20.531301    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd" podUID="69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c" containerName="istio-proxy" containerID="cri-o://396de444acf838c03c82bfca70c53ce0a870ed2a8fa9b81a90a1338f413700c0" gracePeriod=30
Oct 31 01:54:20.540270 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:20.540239    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"]
Oct 31 01:54:20.540518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:20.540492    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj" podUID="2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2" containerName="istio-proxy" containerID="cri-o://03f83f577d9adbf2dffc7bac4ebd0e77bd9171909245b74d40cd6d90247f3355" gracePeriod=30
Oct 31 01:54:23.859686 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:23.859647    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4"
Oct 31 01:54:23.860173 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:23.860145    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv"
Oct 31 01:54:24.411508 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:24.411467    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"]
Oct 31 01:54:24.411715 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:24.411691    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx" podUID="0fd2bc78-97f2-4946-9a5d-f4952c4b49c4" containerName="istio-proxy" containerID="cri-o://0ad9fdc5f00e0e9cd4ed97ef402673832ad221ffc3b9ba4b49b649b556fd560d" gracePeriod=30
Oct 31 01:54:25.986297 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:25.986250    5650 generic.go:356] "Generic (PLEG): container finished" podID="2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2" containerID="03f83f577d9adbf2dffc7bac4ebd0e77bd9171909245b74d40cd6d90247f3355" exitCode=0
Oct 31 01:54:25.987215 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:25.987180    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj" event={"ID":"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2","Type":"ContainerDied","Data":"03f83f577d9adbf2dffc7bac4ebd0e77bd9171909245b74d40cd6d90247f3355"}
Oct 31 01:54:25.992052 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:25.992021    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd" event={"ID":"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c","Type":"ContainerDied","Data":"396de444acf838c03c82bfca70c53ce0a870ed2a8fa9b81a90a1338f413700c0"}
Oct 31 01:54:25.992155 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:25.991757    5650 generic.go:356] "Generic (PLEG): container finished" podID="69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c" containerID="396de444acf838c03c82bfca70c53ce0a870ed2a8fa9b81a90a1338f413700c0" exitCode=0
Oct 31 01:54:26.057680 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.057641    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:54:26.188512 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.180474    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-workload-socket\") pod \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") "
Oct 31 01:54:26.188512 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.180648    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-data\") pod \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") "
Oct 31 01:54:26.188512 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.180687    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-workload-certs\") pod \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") "
Oct 31 01:54:26.188512 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.180756    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-kube-api-access-cwkpj\" (UniqueName: \"kubernetes.io/projected/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-kube-api-access-cwkpj\") pod \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") "
Oct 31 01:54:26.188512 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.180918    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-envoy\") pod \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") "
Oct 31 01:54:26.188512 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.180985    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istiod-ca-cert\") pod \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") "
Oct 31 01:54:26.188512 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.181053    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-podinfo\") pod \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") "
Oct 31 01:54:26.188512 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.181124    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-token\" (UniqueName: \"kubernetes.io/projected/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-token\") pod \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") "
Oct 31 01:54:26.188512 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.181192    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-credential-socket\") pod \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\" (UID: \"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2\") "
Oct 31 01:54:26.188512 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.182368    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-credential-socket" pod "2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2" (UID: "2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.188512 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.182699    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-workload-socket" pod "2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2" (UID: "2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.188512 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.182914    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-data" pod "2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2" (UID: "2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.188512 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.183095    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-workload-certs" pod "2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2" (UID: "2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.188512 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.187394    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istiod-ca-cert" pod "2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2" (UID: "2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:26.195693 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.195641    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-podinfo" pod "2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2" (UID: "2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:54:26.197951 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.197902    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-envoy" pod "2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2" (UID: "2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.212754 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.210803    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-token" pod "2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2" (UID: "2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:26.212754 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.212691    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-kube-api-access-cwkpj" pod "2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2" (UID: "2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2"). InnerVolumeSpecName "kube-api-access-cwkpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:26.237674 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.237616    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs"]
Oct 31 01:54:26.239620 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.239572    5650 cpu_manager.go:414] "containerMap: removing container" podUID="2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2" containerName="istio-proxy"
Oct 31 01:54:26.239816 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.239799    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2" containerName="istio-proxy"
Oct 31 01:54:26.240105 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.240085    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2" containerName="istio-proxy"
Oct 31 01:54:26.246272 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.246249    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs"
Oct 31 01:54:26.254197 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.254083    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:54:26.254197 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.254157    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:54:26.260073 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.259657    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:54:26.294973 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.294894    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-credential-socket\") pod \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") "
Oct 31 01:54:26.295461 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.295397    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-credential-socket" pod "69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c" (UID: "69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.295629 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.295594    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istiod-ca-cert\") pod \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") "
Oct 31 01:54:26.295718 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.295702    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-data\") pod \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") "
Oct 31 01:54:26.295834 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.295817    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-workload-socket\") pod \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") "
Oct 31 01:54:26.295918 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.295884    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-kube-api-access-glswb\" (UniqueName: \"kubernetes.io/projected/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-kube-api-access-glswb\") pod \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") "
Oct 31 01:54:26.295997 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.295976    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-token\" (UniqueName: \"kubernetes.io/projected/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-token\") pod \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") "
Oct 31 01:54:26.296097 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.296067    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-podinfo\") pod \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") "
Oct 31 01:54:26.296186 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.296168    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-envoy\") pod \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") "
Oct 31 01:54:26.296243 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.296212    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-workload-certs\") pod \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\" (UID: \"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c\") "
Oct 31 01:54:26.296733 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.296706    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/232f1f60-5d56-42eb-abcf-a657d20065fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs\" (UID: \"232f1f60-5d56-42eb-abcf-a657d20065fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs"
Oct 31 01:54:26.297053 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.297005    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/232f1f60-5d56-42eb-abcf-a657d20065fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs\" (UID: \"232f1f60-5d56-42eb-abcf-a657d20065fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs"
Oct 31 01:54:26.297168 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.297090    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgmj6\" (UniqueName: \"kubernetes.io/projected/232f1f60-5d56-42eb-abcf-a657d20065fd-kube-api-access-pgmj6\") pod \"maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs\" (UID: \"232f1f60-5d56-42eb-abcf-a657d20065fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs"
Oct 31 01:54:26.297295 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.297277    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.297361 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.297315    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.297361 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.297331    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.297361 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.297346    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwkpj\" (UniqueName: \"kubernetes.io/projected/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-kube-api-access-cwkpj\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.297562 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.297363    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.297562 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.297388    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.297562 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.297402    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.297562 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.297417    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.297562 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.297431    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.297562 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.297494    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.297977 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.297954    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istiod-ca-cert" pod "69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c" (UID: "69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:26.298397 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.298253    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-data" pod "69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c" (UID: "69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.298520 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.298471    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-workload-socket" pod "69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c" (UID: "69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.307851 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.307807    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-workload-certs" pod "69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c" (UID: "69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.329690 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.329646    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs"]
Oct 31 01:54:26.336121 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.336075    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-kube-api-access-glswb" pod "69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c" (UID: "69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c"). InnerVolumeSpecName "kube-api-access-glswb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:26.338674 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.338642    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-token" pod "69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c" (UID: "69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:26.339001 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.338967    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-envoy" pod "69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c" (UID: "69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.343328 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.343297    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-podinfo" pod "69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c" (UID: "69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:54:26.399533 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.399223    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/232f1f60-5d56-42eb-abcf-a657d20065fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs\" (UID: \"232f1f60-5d56-42eb-abcf-a657d20065fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs"
Oct 31 01:54:26.399533 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.399388    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/232f1f60-5d56-42eb-abcf-a657d20065fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs\" (UID: \"232f1f60-5d56-42eb-abcf-a657d20065fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs"
Oct 31 01:54:26.399533 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.399437    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgmj6\" (UniqueName: \"kubernetes.io/projected/232f1f60-5d56-42eb-abcf-a657d20065fd-kube-api-access-pgmj6\") pod \"maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs\" (UID: \"232f1f60-5d56-42eb-abcf-a657d20065fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs"
Oct 31 01:54:26.399830 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.399561    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.399830 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.399579    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.399830 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.399595    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glswb\" (UniqueName: \"kubernetes.io/projected/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-kube-api-access-glswb\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.399830 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.399610    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.399830 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.399624    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.399830 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.399638    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.399830 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.399651    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.399830 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.399664    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.402788 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.401332    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/232f1f60-5d56-42eb-abcf-a657d20065fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs\" (UID: \"232f1f60-5d56-42eb-abcf-a657d20065fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs"
Oct 31 01:54:26.404234 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.404155    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/232f1f60-5d56-42eb-abcf-a657d20065fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs\" (UID: \"232f1f60-5d56-42eb-abcf-a657d20065fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs"
Oct 31 01:54:26.416279 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.416248    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgmj6\" (UniqueName: \"kubernetes.io/projected/232f1f60-5d56-42eb-abcf-a657d20065fd-kube-api-access-pgmj6\") pod \"maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs\" (UID: \"232f1f60-5d56-42eb-abcf-a657d20065fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs"
Oct 31 01:54:26.637105 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:26.637059    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs"
Oct 31 01:54:27.548133 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:27.548028    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"
Oct 31 01:54:27.609809 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:27.548599    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd" event={"ID":"69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c","Type":"ContainerDied","Data":"2584a7802f015b39b7a5553b44de782fa2973b64125a5db68c59a60ee3cedda1"}
Oct 31 01:54:27.610132 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:27.610104    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs"]
Oct 31 01:54:27.613471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:27.610291    5650 scope.go:122] "RemoveContainer" containerID="396de444acf838c03c82bfca70c53ce0a870ed2a8fa9b81a90a1338f413700c0"
Oct 31 01:54:27.671409 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:27.671375    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj" event={"ID":"2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2","Type":"ContainerDied","Data":"9c776b58998f2eac1401a3010dbb4fe94625d8631da3562755862b0ccbfc5443"}
Oct 31 01:54:27.673476 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:27.673411    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"
Oct 31 01:54:27.724048 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:27.724015    5650 scope.go:122] "RemoveContainer" containerID="03f83f577d9adbf2dffc7bac4ebd0e77bd9171909245b74d40cd6d90247f3355"
Oct 31 01:54:27.822918 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:27.822667    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"]
Oct 31 01:54:27.853629 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:27.853587    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:27.890562 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:27.890517    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:27.900078 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:27.900029    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:27.949558 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:27.949480    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-9wstd"]
Oct 31 01:54:27.955958 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:27.955926    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"]
Oct 31 01:54:27.962319 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:27.962277    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-p6jpj"]
Oct 31 01:54:28.056471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:28.051586    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:28.143638 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:28.143323    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:28.238464 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:28.215402    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2" path="/var/lib/kubelet/pods/2bb3cc6b-e51d-4d13-ab21-572d0cd4dfb2/volumes"
Oct 31 01:54:28.238464 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:28.235781    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c" path="/var/lib/kubelet/pods/69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c/volumes"
Oct 31 01:54:28.385469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:28.324589    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 01:54:28.517948 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:28.513566    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 01:54:28.731533 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:28.706252    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs" event={"ID":"232f1f60-5d56-42eb-abcf-a657d20065fd","Type":"ContainerStarted","Data":"8ae041faa12b5b9f08de5ae0c877f165b348e7e7ae8f3f8ecd0f4a316e1681c6"}
Oct 31 01:54:28.731533 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:28.706363    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs" event={"ID":"232f1f60-5d56-42eb-abcf-a657d20065fd","Type":"ContainerStarted","Data":"ec745f85447f14c9bb86fba9a4bfd037f0e57203607df29785f5c165668c0a5d"}
Oct 31 01:54:28.731533 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:28.729203    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs" podStartSLOduration=2.729184391 podStartE2EDuration="2.729184391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:28.728982571 +0000 UTC m=+2024.732696865" watchObservedRunningTime="2026-10-31 01:54:28.729184391 +0000 UTC m=+2024.732898683"
Oct 31 01:54:28.821527 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:54:28.817584    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:54:29.749414 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.749372    5650 generic.go:356] "Generic (PLEG): container finished" podID="0fd2bc78-97f2-4946-9a5d-f4952c4b49c4" containerID="0ad9fdc5f00e0e9cd4ed97ef402673832ad221ffc3b9ba4b49b649b556fd560d" exitCode=0
Oct 31 01:54:29.750006 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.749474    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx" event={"ID":"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4","Type":"ContainerDied","Data":"0ad9fdc5f00e0e9cd4ed97ef402673832ad221ffc3b9ba4b49b649b556fd560d"}
Oct 31 01:54:29.750006 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.749539    5650 scope.go:122] "RemoveContainer" containerID="4b53119a1a231d0eaad1f41d77afdbe6ec0e779fd49ec6cbd7648008aeecdb8a"
Oct 31 01:54:29.751851 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.751767    5650 generic.go:356] "Generic (PLEG): container finished" podID="232f1f60-5d56-42eb-abcf-a657d20065fd" containerID="8ae041faa12b5b9f08de5ae0c877f165b348e7e7ae8f3f8ecd0f4a316e1681c6" exitCode=0
Oct 31 01:54:29.751934 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.751913    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs" event={"ID":"232f1f60-5d56-42eb-abcf-a657d20065fd","Type":"ContainerDied","Data":"8ae041faa12b5b9f08de5ae0c877f165b348e7e7ae8f3f8ecd0f4a316e1681c6"}
Oct 31 01:54:29.808578 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.808536    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:54:29.899813 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.899779    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istiod-ca-cert\") pod \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") "
Oct 31 01:54:29.899994 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.899824    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-workload-socket\") pod \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") "
Oct 31 01:54:29.899994 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.899873    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-podinfo\") pod \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") "
Oct 31 01:54:29.900075 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.900045    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-credential-socket\") pod \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") "
Oct 31 01:54:29.900151 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.900129    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-kube-api-access-mhtn7\" (UniqueName: \"kubernetes.io/projected/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-kube-api-access-mhtn7\") pod \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") "
Oct 31 01:54:29.900217 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.900164    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-workload-socket" pod "0fd2bc78-97f2-4946-9a5d-f4952c4b49c4" (UID: "0fd2bc78-97f2-4946-9a5d-f4952c4b49c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:29.900271 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.900226    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-data\") pod \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") "
Oct 31 01:54:29.900271 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.900234    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istiod-ca-cert" pod "0fd2bc78-97f2-4946-9a5d-f4952c4b49c4" (UID: "0fd2bc78-97f2-4946-9a5d-f4952c4b49c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:29.900382 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.900331    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-credential-socket" pod "0fd2bc78-97f2-4946-9a5d-f4952c4b49c4" (UID: "0fd2bc78-97f2-4946-9a5d-f4952c4b49c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:29.900382 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.900347    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-token\" (UniqueName: \"kubernetes.io/projected/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-token\") pod \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") "
Oct 31 01:54:29.900489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.900392    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-envoy\") pod \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") "
Oct 31 01:54:29.900550 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.900526    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-workload-certs\") pod \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\" (UID: \"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4\") "
Oct 31 01:54:29.900610 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.900549    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-data" pod "0fd2bc78-97f2-4946-9a5d-f4952c4b49c4" (UID: "0fd2bc78-97f2-4946-9a5d-f4952c4b49c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:29.900872 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.900847    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-workload-certs" pod "0fd2bc78-97f2-4946-9a5d-f4952c4b49c4" (UID: "0fd2bc78-97f2-4946-9a5d-f4952c4b49c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:29.901314 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.901292    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:29.901422 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.901318    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:29.901422 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.901333    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:29.901422 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.901348    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:29.901422 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.901362    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:29.902608 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.902552    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-token" pod "0fd2bc78-97f2-4946-9a5d-f4952c4b49c4" (UID: "0fd2bc78-97f2-4946-9a5d-f4952c4b49c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:29.902608 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.902556    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-podinfo" pod "0fd2bc78-97f2-4946-9a5d-f4952c4b49c4" (UID: "0fd2bc78-97f2-4946-9a5d-f4952c4b49c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:54:29.902809 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.902790    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-envoy" pod "0fd2bc78-97f2-4946-9a5d-f4952c4b49c4" (UID: "0fd2bc78-97f2-4946-9a5d-f4952c4b49c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:29.902988 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:29.902970    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-kube-api-access-mhtn7" pod "0fd2bc78-97f2-4946-9a5d-f4952c4b49c4" (UID: "0fd2bc78-97f2-4946-9a5d-f4952c4b49c4"). InnerVolumeSpecName "kube-api-access-mhtn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:30.003229 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:30.003125    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:30.003229 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:30.003170    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhtn7\" (UniqueName: \"kubernetes.io/projected/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-kube-api-access-mhtn7\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:30.003229 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:30.003189    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:30.003229 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:30.003203    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:30.760704 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:30.760664    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"
Oct 31 01:54:30.761165 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:30.760735    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx" event={"ID":"0fd2bc78-97f2-4946-9a5d-f4952c4b49c4","Type":"ContainerDied","Data":"faab513cb79265896f1d893f6ebaf97274720a41775f734cba1486cdaca93c23"}
Oct 31 01:54:30.761165 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:30.760927    5650 scope.go:122] "RemoveContainer" containerID="0ad9fdc5f00e0e9cd4ed97ef402673832ad221ffc3b9ba4b49b649b556fd560d"
Oct 31 01:54:30.790389 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:30.790353    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"]
Oct 31 01:54:30.794066 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:30.794036    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-5f65e1-openshift-default-5999bfbc6f-8skgx"]
Oct 31 01:54:30.951828 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:30.951797    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs"
Oct 31 01:54:31.018420 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:31.018315    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/232f1f60-5d56-42eb-abcf-a657d20065fd-kube-api-access-pgmj6\" (UniqueName: \"kubernetes.io/projected/232f1f60-5d56-42eb-abcf-a657d20065fd-kube-api-access-pgmj6\") pod \"232f1f60-5d56-42eb-abcf-a657d20065fd\" (UID: \"232f1f60-5d56-42eb-abcf-a657d20065fd\") "
Oct 31 01:54:31.018628 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:31.018608    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/232f1f60-5d56-42eb-abcf-a657d20065fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/232f1f60-5d56-42eb-abcf-a657d20065fd-maas-api-openshift-ca\") pod \"232f1f60-5d56-42eb-abcf-a657d20065fd\" (UID: \"232f1f60-5d56-42eb-abcf-a657d20065fd\") "
Oct 31 01:54:31.018694 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:31.018670    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/232f1f60-5d56-42eb-abcf-a657d20065fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/232f1f60-5d56-42eb-abcf-a657d20065fd-maas-api-xks-ca\") pod \"232f1f60-5d56-42eb-abcf-a657d20065fd\" (UID: \"232f1f60-5d56-42eb-abcf-a657d20065fd\") "
Oct 31 01:54:31.019126 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:31.019098    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/232f1f60-5d56-42eb-abcf-a657d20065fd-maas-api-openshift-ca" pod "232f1f60-5d56-42eb-abcf-a657d20065fd" (UID: "232f1f60-5d56-42eb-abcf-a657d20065fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:31.019327 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:31.019299    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/232f1f60-5d56-42eb-abcf-a657d20065fd-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:31.020773 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:31.020746    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232f1f60-5d56-42eb-abcf-a657d20065fd-kube-api-access-pgmj6" pod "232f1f60-5d56-42eb-abcf-a657d20065fd" (UID: "232f1f60-5d56-42eb-abcf-a657d20065fd"). InnerVolumeSpecName "kube-api-access-pgmj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:31.021231 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:31.021215    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/232f1f60-5d56-42eb-abcf-a657d20065fd-maas-api-xks-ca" pod "232f1f60-5d56-42eb-abcf-a657d20065fd" (UID: "232f1f60-5d56-42eb-abcf-a657d20065fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:54:31.120748 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:31.120707    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/232f1f60-5d56-42eb-abcf-a657d20065fd-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:31.120748 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:31.120751    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgmj6\" (UniqueName: \"kubernetes.io/projected/232f1f60-5d56-42eb-abcf-a657d20065fd-kube-api-access-pgmj6\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:31.823470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:31.822994    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs"
Oct 31 01:54:31.830565 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:31.828395    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs" event={"ID":"232f1f60-5d56-42eb-abcf-a657d20065fd","Type":"ContainerDied","Data":"ec745f85447f14c9bb86fba9a4bfd037f0e57203607df29785f5c165668c0a5d"}
Oct 31 01:54:31.830565 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:31.828477    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec745f85447f14c9bb86fba9a4bfd037f0e57203607df29785f5c165668c0a5d"
Oct 31 01:54:32.456298 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.450534    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fd2bc78-97f2-4946-9a5d-f4952c4b49c4" path="/var/lib/kubelet/pods/0fd2bc78-97f2-4946-9a5d-f4952c4b49c4/volumes"
Oct 31 01:54:32.456298 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.451106    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"]
Oct 31 01:54:32.456298 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.451947    5650 cpu_manager.go:414] "containerMap: removing container" podUID="69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c" containerName="istio-proxy"
Oct 31 01:54:32.456298 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.451962    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c" containerName="istio-proxy"
Oct 31 01:54:32.456298 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.451978    5650 cpu_manager.go:414] "containerMap: removing container" podUID="232f1f60-5d56-42eb-abcf-a657d20065fd" containerName="revoke-keys"
Oct 31 01:54:32.456298 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.451986    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="232f1f60-5d56-42eb-abcf-a657d20065fd" containerName="revoke-keys"
Oct 31 01:54:32.456298 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.452011    5650 cpu_manager.go:414] "containerMap: removing container" podUID="0fd2bc78-97f2-4946-9a5d-f4952c4b49c4" containerName="istio-proxy"
Oct 31 01:54:32.456298 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.452019    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd2bc78-97f2-4946-9a5d-f4952c4b49c4" containerName="istio-proxy"
Oct 31 01:54:32.456298 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.452064    5650 cpu_manager.go:414] "containerMap: removing container" podUID="0fd2bc78-97f2-4946-9a5d-f4952c4b49c4" containerName="istio-proxy"
Oct 31 01:54:32.456298 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.452071    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd2bc78-97f2-4946-9a5d-f4952c4b49c4" containerName="istio-proxy"
Oct 31 01:54:32.456298 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.452209    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd2bc78-97f2-4946-9a5d-f4952c4b49c4" containerName="istio-proxy"
Oct 31 01:54:32.456298 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.452225    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="69d0d6a5-b1d9-41ae-9e5b-a41abc3ce02c" containerName="istio-proxy"
Oct 31 01:54:32.456298 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.452236    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="232f1f60-5d56-42eb-abcf-a657d20065fd" containerName="revoke-keys"
Oct 31 01:54:32.490492 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.490376    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"]
Oct 31 01:54:32.490678 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.490513    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.498580 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.496703    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-dbff33-openshift-default-dockercfg-w5k8r\""
Oct 31 01:54:32.590120 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.590072    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6011af8-136f-436a-bb13-987d5844f336-istiod-ca-cert\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.590314 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.590125    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-credential-socket\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.590314 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.590285    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-workload-socket\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.590473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.590359    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6011af8-136f-436a-bb13-987d5844f336-istio-podinfo\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.590568 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.590545    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6011af8-136f-436a-bb13-987d5844f336-istio-token\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.590654 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.590639    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-workload-certs\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.590698 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.590672    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-istio-data\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.590849 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.590828    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck2jv\" (UniqueName: \"kubernetes.io/projected/f6011af8-136f-436a-bb13-987d5844f336-kube-api-access-ck2jv\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.590991 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.590977    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-istio-envoy\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.696840 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.695457    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-istio-envoy\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.696840 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.695538    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6011af8-136f-436a-bb13-987d5844f336-istiod-ca-cert\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.696840 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.695570    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-credential-socket\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.696840 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.695613    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-workload-socket\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.696840 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.695671    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6011af8-136f-436a-bb13-987d5844f336-istio-podinfo\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.696840 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.695742    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6011af8-136f-436a-bb13-987d5844f336-istio-token\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.696840 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.695805    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-workload-certs\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.696840 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.695839    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-istio-data\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.696840 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.695946    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck2jv\" (UniqueName: \"kubernetes.io/projected/f6011af8-136f-436a-bb13-987d5844f336-kube-api-access-ck2jv\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.697837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.697803    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-credential-socket\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.697950 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.697819    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-workload-certs\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.698128 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.698101    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-istio-data\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.698296 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.698250    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6011af8-136f-436a-bb13-987d5844f336-istiod-ca-cert\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.698430 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.698293    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-workload-socket\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.699031 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.699008    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-istio-envoy\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.700633 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.700606    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6011af8-136f-436a-bb13-987d5844f336-istio-podinfo\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.710013 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.709938    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck2jv\" (UniqueName: \"kubernetes.io/projected/f6011af8-136f-436a-bb13-987d5844f336-kube-api-access-ck2jv\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.711300 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.711277    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6011af8-136f-436a-bb13-987d5844f336-istio-token\") pod \"e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") " pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.812524 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.812469    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:32.915498 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.914636    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bae1d5ce-8mx2n"]
Oct 31 01:54:32.915498 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.915416    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd2bc78-97f2-4946-9a5d-f4952c4b49c4" containerName="istio-proxy"
Oct 31 01:54:32.921584 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.921350    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bae1d5ce-8mx2n"
Oct 31 01:54:32.924490 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.923979    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bae1d5ce-8mx2n"]
Oct 31 01:54:32.971628 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.971523    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:54:32.971810 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:32.971642    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:54:33.002578 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:33.002542    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qssg\" (UniqueName: \"kubernetes.io/projected/758dae21-5a65-4610-a7e9-9fd29f6021a9-kube-api-access-6qssg\") pod \"maas-api-revoke-sub-bae1d5ce-8mx2n\" (UID: \"758dae21-5a65-4610-a7e9-9fd29f6021a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bae1d5ce-8mx2n"
Oct 31 01:54:33.002578 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:33.002593    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758dae21-5a65-4610-a7e9-9fd29f6021a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bae1d5ce-8mx2n\" (UID: \"758dae21-5a65-4610-a7e9-9fd29f6021a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bae1d5ce-8mx2n"
Oct 31 01:54:33.002943 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:33.002738    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758dae21-5a65-4610-a7e9-9fd29f6021a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bae1d5ce-8mx2n\" (UID: \"758dae21-5a65-4610-a7e9-9fd29f6021a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bae1d5ce-8mx2n"
Oct 31 01:54:33.103717 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:33.103685    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758dae21-5a65-4610-a7e9-9fd29f6021a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bae1d5ce-8mx2n\" (UID: \"758dae21-5a65-4610-a7e9-9fd29f6021a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bae1d5ce-8mx2n"
Oct 31 01:54:33.103885 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:33.103829    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qssg\" (UniqueName: \"kubernetes.io/projected/758dae21-5a65-4610-a7e9-9fd29f6021a9-kube-api-access-6qssg\") pod \"maas-api-revoke-sub-bae1d5ce-8mx2n\" (UID: \"758dae21-5a65-4610-a7e9-9fd29f6021a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bae1d5ce-8mx2n"
Oct 31 01:54:33.103885 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:33.103863    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758dae21-5a65-4610-a7e9-9fd29f6021a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bae1d5ce-8mx2n\" (UID: \"758dae21-5a65-4610-a7e9-9fd29f6021a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bae1d5ce-8mx2n"
Oct 31 01:54:33.104531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:33.104504    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758dae21-5a65-4610-a7e9-9fd29f6021a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bae1d5ce-8mx2n\" (UID: \"758dae21-5a65-4610-a7e9-9fd29f6021a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bae1d5ce-8mx2n"
Oct 31 01:54:33.106545 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:33.106520    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758dae21-5a65-4610-a7e9-9fd29f6021a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bae1d5ce-8mx2n\" (UID: \"758dae21-5a65-4610-a7e9-9fd29f6021a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bae1d5ce-8mx2n"
Oct 31 01:54:33.118797 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:54:33.118751    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6011af8_136f_436a_bb13_987d5844f336.slice/crio-8be05334a3858f48e63eb9c14176d7c37417d5c4915d2bf8d49a393e27e47395 WatchSource:0}: Error finding container 8be05334a3858f48e63eb9c14176d7c37417d5c4915d2bf8d49a393e27e47395: Status 404 returned error can't find the container with id 8be05334a3858f48e63eb9c14176d7c37417d5c4915d2bf8d49a393e27e47395
Oct 31 01:54:33.119554 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:33.119517    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"]
Oct 31 01:54:33.120756 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:33.120735    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qssg\" (UniqueName: \"kubernetes.io/projected/758dae21-5a65-4610-a7e9-9fd29f6021a9-kube-api-access-6qssg\") pod \"maas-api-revoke-sub-bae1d5ce-8mx2n\" (UID: \"758dae21-5a65-4610-a7e9-9fd29f6021a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bae1d5ce-8mx2n"
Oct 31 01:54:33.121951 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:33.121922    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:54:33.122010 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:33.121984    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:54:33.122056 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:33.122012    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:54:33.248868 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:33.248772    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bae1d5ce-8mx2n"
Oct 31 01:54:33.541377 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:54:33.541343    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod758dae21_5a65_4610_a7e9_9fd29f6021a9.slice/crio-020570afd2e04a1f17060fe22bb43eb8e2c95761d9d52ab3ac57d1da03ad9f21 WatchSource:0}: Error finding container 020570afd2e04a1f17060fe22bb43eb8e2c95761d9d52ab3ac57d1da03ad9f21: Status 404 returned error can't find the container with id 020570afd2e04a1f17060fe22bb43eb8e2c95761d9d52ab3ac57d1da03ad9f21
Oct 31 01:54:33.541542 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:33.541484    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bae1d5ce-8mx2n"]
Oct 31 01:54:33.850851 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:33.850806    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bae1d5ce-8mx2n" event={"ID":"758dae21-5a65-4610-a7e9-9fd29f6021a9","Type":"ContainerStarted","Data":"88212fe5ef44b7ec845c18c2ad65e53910b67d16826ddbb662f383b47d0bac8a"}
Oct 31 01:54:33.851055 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:33.850875    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bae1d5ce-8mx2n" event={"ID":"758dae21-5a65-4610-a7e9-9fd29f6021a9","Type":"ContainerStarted","Data":"020570afd2e04a1f17060fe22bb43eb8e2c95761d9d52ab3ac57d1da03ad9f21"}
Oct 31 01:54:33.852854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:33.852824    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z" event={"ID":"f6011af8-136f-436a-bb13-987d5844f336","Type":"ContainerStarted","Data":"bbba164712e42fa94a9c7cc2065cc488af9e37267c86017d4c457dfc8af744fa"}
Oct 31 01:54:33.852955 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:33.852880    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z" event={"ID":"f6011af8-136f-436a-bb13-987d5844f336","Type":"ContainerStarted","Data":"8be05334a3858f48e63eb9c14176d7c37417d5c4915d2bf8d49a393e27e47395"}
Oct 31 01:54:33.873231 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:33.873172    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bae1d5ce-8mx2n" podStartSLOduration=1.873143288 podStartE2EDuration="1.873143288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:33.87024432 +0000 UTC m=+2029.873958612" watchObservedRunningTime="2026-10-31 01:54:33.873143288 +0000 UTC m=+2029.876857579"
Oct 31 01:54:33.900357 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:33.900301    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z" podStartSLOduration=1.900286615 podStartE2EDuration="1.900286615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:33.89836404 +0000 UTC m=+2029.902078332" watchObservedRunningTime="2026-10-31 01:54:33.900286615 +0000 UTC m=+2029.904000968"
Oct 31 01:54:34.813570 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:34.813526    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:34.819743 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:34.819715    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:34.862660 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:34.862626    5650 generic.go:356] "Generic (PLEG): container finished" podID="758dae21-5a65-4610-a7e9-9fd29f6021a9" containerID="88212fe5ef44b7ec845c18c2ad65e53910b67d16826ddbb662f383b47d0bac8a" exitCode=0
Oct 31 01:54:34.862833 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:34.862704    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bae1d5ce-8mx2n" event={"ID":"758dae21-5a65-4610-a7e9-9fd29f6021a9","Type":"ContainerDied","Data":"88212fe5ef44b7ec845c18c2ad65e53910b67d16826ddbb662f383b47d0bac8a"}
Oct 31 01:54:34.863070 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:34.863053    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:34.864060 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:34.864040    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:54:36.029725 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:36.029698    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bae1d5ce-8mx2n"
Oct 31 01:54:36.156378 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:36.156276    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/758dae21-5a65-4610-a7e9-9fd29f6021a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758dae21-5a65-4610-a7e9-9fd29f6021a9-maas-api-openshift-ca\") pod \"758dae21-5a65-4610-a7e9-9fd29f6021a9\" (UID: \"758dae21-5a65-4610-a7e9-9fd29f6021a9\") "
Oct 31 01:54:36.156605 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:36.156480    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/758dae21-5a65-4610-a7e9-9fd29f6021a9-kube-api-access-6qssg\" (UniqueName: \"kubernetes.io/projected/758dae21-5a65-4610-a7e9-9fd29f6021a9-kube-api-access-6qssg\") pod \"758dae21-5a65-4610-a7e9-9fd29f6021a9\" (UID: \"758dae21-5a65-4610-a7e9-9fd29f6021a9\") "
Oct 31 01:54:36.156605 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:36.156536    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/758dae21-5a65-4610-a7e9-9fd29f6021a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758dae21-5a65-4610-a7e9-9fd29f6021a9-maas-api-xks-ca\") pod \"758dae21-5a65-4610-a7e9-9fd29f6021a9\" (UID: \"758dae21-5a65-4610-a7e9-9fd29f6021a9\") "
Oct 31 01:54:36.158026 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:36.156755    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/758dae21-5a65-4610-a7e9-9fd29f6021a9-maas-api-openshift-ca" pod "758dae21-5a65-4610-a7e9-9fd29f6021a9" (UID: "758dae21-5a65-4610-a7e9-9fd29f6021a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:36.158026 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:36.157247    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758dae21-5a65-4610-a7e9-9fd29f6021a9-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:36.159395 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:36.159366    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758dae21-5a65-4610-a7e9-9fd29f6021a9-kube-api-access-6qssg" pod "758dae21-5a65-4610-a7e9-9fd29f6021a9" (UID: "758dae21-5a65-4610-a7e9-9fd29f6021a9"). InnerVolumeSpecName "kube-api-access-6qssg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:36.159533 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:36.159488    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/758dae21-5a65-4610-a7e9-9fd29f6021a9-maas-api-xks-ca" pod "758dae21-5a65-4610-a7e9-9fd29f6021a9" (UID: "758dae21-5a65-4610-a7e9-9fd29f6021a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:54:36.259514 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:36.259475    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qssg\" (UniqueName: \"kubernetes.io/projected/758dae21-5a65-4610-a7e9-9fd29f6021a9-kube-api-access-6qssg\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:36.259514 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:36.259509    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758dae21-5a65-4610-a7e9-9fd29f6021a9-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:36.395032 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:36.394985    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc"]
Oct 31 01:54:36.415169 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:36.415080    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q"]
Oct 31 01:54:36.879373 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:36.879322    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bae1d5ce-8mx2n" event={"ID":"758dae21-5a65-4610-a7e9-9fd29f6021a9","Type":"ContainerDied","Data":"020570afd2e04a1f17060fe22bb43eb8e2c95761d9d52ab3ac57d1da03ad9f21"}
Oct 31 01:54:36.879727 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:36.879397    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="020570afd2e04a1f17060fe22bb43eb8e2c95761d9d52ab3ac57d1da03ad9f21"
Oct 31 01:54:36.879727 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:36.879604    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bae1d5ce-8mx2n"
Oct 31 01:54:37.471632 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:37.471586    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh"]
Oct 31 01:54:37.472718 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:37.472690    5650 cpu_manager.go:414] "containerMap: removing container" podUID="758dae21-5a65-4610-a7e9-9fd29f6021a9" containerName="revoke-keys"
Oct 31 01:54:37.472718 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:37.472718    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="758dae21-5a65-4610-a7e9-9fd29f6021a9" containerName="revoke-keys"
Oct 31 01:54:37.472965 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:37.472953    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="758dae21-5a65-4610-a7e9-9fd29f6021a9" containerName="revoke-keys"
Oct 31 01:54:37.476690 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:37.476666    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh"
Oct 31 01:54:37.484907 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:37.484875    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh"]
Oct 31 01:54:37.487821 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:37.487792    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-dbff33\""
Oct 31 01:54:37.590903 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:37.590862    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th96c\" (UniqueName: \"kubernetes.io/projected/26b7a731-46d2-4702-94c3-56ed4ad79399-kube-api-access-th96c\") pod \"payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh\" (UID: \"26b7a731-46d2-4702-94c3-56ed4ad79399\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh"
Oct 31 01:54:37.590903 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:37.590908    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b7a731-46d2-4702-94c3-56ed4ad79399-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh\" (UID: \"26b7a731-46d2-4702-94c3-56ed4ad79399\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh"
Oct 31 01:54:37.695937 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:37.695891    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th96c\" (UniqueName: \"kubernetes.io/projected/26b7a731-46d2-4702-94c3-56ed4ad79399-kube-api-access-th96c\") pod \"payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh\" (UID: \"26b7a731-46d2-4702-94c3-56ed4ad79399\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh"
Oct 31 01:54:37.696157 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:37.695966    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b7a731-46d2-4702-94c3-56ed4ad79399-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh\" (UID: \"26b7a731-46d2-4702-94c3-56ed4ad79399\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh"
Oct 31 01:54:37.719485 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:37.712155    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b7a731-46d2-4702-94c3-56ed4ad79399-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh\" (UID: \"26b7a731-46d2-4702-94c3-56ed4ad79399\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh"
Oct 31 01:54:37.719485 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:37.715920    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th96c\" (UniqueName: \"kubernetes.io/projected/26b7a731-46d2-4702-94c3-56ed4ad79399-kube-api-access-th96c\") pod \"payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh\" (UID: \"26b7a731-46d2-4702-94c3-56ed4ad79399\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh"
Oct 31 01:54:37.803397 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:37.799631    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh"
Oct 31 01:54:38.165043 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:54:38.164987    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b7a731_46d2_4702_94c3_56ed4ad79399.slice/crio-6cd4f26d1075d6413b0fcff2bbbff7e75cd829b35e0cd10ac0e33520fa3f1d34 WatchSource:0}: Error finding container 6cd4f26d1075d6413b0fcff2bbbff7e75cd829b35e0cd10ac0e33520fa3f1d34: Status 404 returned error can't find the container with id 6cd4f26d1075d6413b0fcff2bbbff7e75cd829b35e0cd10ac0e33520fa3f1d34
Oct 31 01:54:38.171391 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:38.170651    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh"]
Oct 31 01:54:38.278122 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:38.277300    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs"]
Oct 31 01:54:38.282531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:38.282504    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-726dd1-6tccs"]
Oct 31 01:54:38.901932 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:38.901883    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh" event={"ID":"26b7a731-46d2-4702-94c3-56ed4ad79399","Type":"ContainerStarted","Data":"ea66690969c56278e1c3ac545d1b8c2eddcd2ee03ebba1dfda555cfd8dea72a1"}
Oct 31 01:54:38.902305 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:38.901949    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh" event={"ID":"26b7a731-46d2-4702-94c3-56ed4ad79399","Type":"ContainerStarted","Data":"6cd4f26d1075d6413b0fcff2bbbff7e75cd829b35e0cd10ac0e33520fa3f1d34"}
Oct 31 01:54:38.902305 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:38.901980    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh"
Oct 31 01:54:38.925538 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:38.925485    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh" podStartSLOduration=1.925470628 podStartE2EDuration="1.925470628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:38.922035541 +0000 UTC m=+2034.925749830" watchObservedRunningTime="2026-10-31 01:54:38.925470628 +0000 UTC m=+2034.929184918"
Oct 31 01:54:40.119973 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:40.119927    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="232f1f60-5d56-42eb-abcf-a657d20065fd" path="/var/lib/kubelet/pods/232f1f60-5d56-42eb-abcf-a657d20065fd/volumes"
Oct 31 01:54:41.396037 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.395988    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc" podUID="58ae8229-51c6-4263-a3b2-ec6616c90d0f" containerName="payload-processing" containerID="cri-o://9bdc3cbb1a0de66f2b6036dc4ac64920a2853329964824fb0814a1a7348c310d" gracePeriod=25
Oct 31 01:54:41.416344 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.416294    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q" podUID="0c759abc-5d22-4c0a-884e-5d99bcf79fdb" containerName="payload-pre-processing" containerID="cri-o://091fc9254cd3926924319c8a44fce8cb689d5b08155c1b5ad8d0ba5fb59c0d66" gracePeriod=25
Oct 31 01:54:41.699188 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.699159    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc"
Oct 31 01:54:41.780396 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.780364    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q"
Oct 31 01:54:41.788384 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.788352    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c759abc-5d22-4c0a-884e-5d99bcf79fdb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c759abc-5d22-4c0a-884e-5d99bcf79fdb-plugins-config-volume\") pod \"0c759abc-5d22-4c0a-884e-5d99bcf79fdb\" (UID: \"0c759abc-5d22-4c0a-884e-5d99bcf79fdb\") "
Oct 31 01:54:41.788588 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.788463    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58ae8229-51c6-4263-a3b2-ec6616c90d0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ae8229-51c6-4263-a3b2-ec6616c90d0f-plugins-config-volume\") pod \"58ae8229-51c6-4263-a3b2-ec6616c90d0f\" (UID: \"58ae8229-51c6-4263-a3b2-ec6616c90d0f\") "
Oct 31 01:54:41.788672 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.788594    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c759abc-5d22-4c0a-884e-5d99bcf79fdb-kube-api-access-xq97d\" (UniqueName: \"kubernetes.io/projected/0c759abc-5d22-4c0a-884e-5d99bcf79fdb-kube-api-access-xq97d\") pod \"0c759abc-5d22-4c0a-884e-5d99bcf79fdb\" (UID: \"0c759abc-5d22-4c0a-884e-5d99bcf79fdb\") "
Oct 31 01:54:41.788735 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.788717    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58ae8229-51c6-4263-a3b2-ec6616c90d0f-kube-api-access-qjp48\" (UniqueName: \"kubernetes.io/projected/58ae8229-51c6-4263-a3b2-ec6616c90d0f-kube-api-access-qjp48\") pod \"58ae8229-51c6-4263-a3b2-ec6616c90d0f\" (UID: \"58ae8229-51c6-4263-a3b2-ec6616c90d0f\") "
Oct 31 01:54:41.788847 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.788827    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c759abc-5d22-4c0a-884e-5d99bcf79fdb-plugins-config-volume" pod "0c759abc-5d22-4c0a-884e-5d99bcf79fdb" (UID: "0c759abc-5d22-4c0a-884e-5d99bcf79fdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:41.788908 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.788862    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ae8229-51c6-4263-a3b2-ec6616c90d0f-plugins-config-volume" pod "58ae8229-51c6-4263-a3b2-ec6616c90d0f" (UID: "58ae8229-51c6-4263-a3b2-ec6616c90d0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:41.789519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.789498    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ae8229-51c6-4263-a3b2-ec6616c90d0f-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:41.789519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.789517    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c759abc-5d22-4c0a-884e-5d99bcf79fdb-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:41.790812 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.790787    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ae8229-51c6-4263-a3b2-ec6616c90d0f-kube-api-access-qjp48" pod "58ae8229-51c6-4263-a3b2-ec6616c90d0f" (UID: "58ae8229-51c6-4263-a3b2-ec6616c90d0f"). InnerVolumeSpecName "kube-api-access-qjp48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:41.790905 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.790821    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c759abc-5d22-4c0a-884e-5d99bcf79fdb-kube-api-access-xq97d" pod "0c759abc-5d22-4c0a-884e-5d99bcf79fdb" (UID: "0c759abc-5d22-4c0a-884e-5d99bcf79fdb"). InnerVolumeSpecName "kube-api-access-xq97d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:41.891362 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.891321    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq97d\" (UniqueName: \"kubernetes.io/projected/0c759abc-5d22-4c0a-884e-5d99bcf79fdb-kube-api-access-xq97d\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:41.891630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.891372    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjp48\" (UniqueName: \"kubernetes.io/projected/58ae8229-51c6-4263-a3b2-ec6616c90d0f-kube-api-access-qjp48\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:41.936678 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.936595    5650 generic.go:356] "Generic (PLEG): container finished" podID="0c759abc-5d22-4c0a-884e-5d99bcf79fdb" containerID="091fc9254cd3926924319c8a44fce8cb689d5b08155c1b5ad8d0ba5fb59c0d66" exitCode=0
Oct 31 01:54:41.936678 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.936668    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q"
Oct 31 01:54:41.936892 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.936681    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q" event={"ID":"0c759abc-5d22-4c0a-884e-5d99bcf79fdb","Type":"ContainerDied","Data":"091fc9254cd3926924319c8a44fce8cb689d5b08155c1b5ad8d0ba5fb59c0d66"}
Oct 31 01:54:41.936892 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.936769    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q" event={"ID":"0c759abc-5d22-4c0a-884e-5d99bcf79fdb","Type":"ContainerDied","Data":"54606dc510d5be237fb7f906c96cdbd49f73b938f4c311bdc5ff9a2ddd5fafee"}
Oct 31 01:54:41.936892 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.936788    5650 scope.go:122] "RemoveContainer" containerID="091fc9254cd3926924319c8a44fce8cb689d5b08155c1b5ad8d0ba5fb59c0d66"
Oct 31 01:54:41.938158 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.938133    5650 generic.go:356] "Generic (PLEG): container finished" podID="58ae8229-51c6-4263-a3b2-ec6616c90d0f" containerID="9bdc3cbb1a0de66f2b6036dc4ac64920a2853329964824fb0814a1a7348c310d" exitCode=0
Oct 31 01:54:41.938239 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.938188    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc"
Oct 31 01:54:41.938239 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.938220    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc" event={"ID":"58ae8229-51c6-4263-a3b2-ec6616c90d0f","Type":"ContainerDied","Data":"9bdc3cbb1a0de66f2b6036dc4ac64920a2853329964824fb0814a1a7348c310d"}
Oct 31 01:54:41.938312 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.938273    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc" event={"ID":"58ae8229-51c6-4263-a3b2-ec6616c90d0f","Type":"ContainerDied","Data":"9a5bab29c89d8153c76b1217a5df29d74f5ccfab4f8ff27bab500e353a38b66f"}
Oct 31 01:54:41.949797 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.949775    5650 scope.go:122] "RemoveContainer" containerID="091fc9254cd3926924319c8a44fce8cb689d5b08155c1b5ad8d0ba5fb59c0d66"
Oct 31 01:54:41.950081 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:54:41.950061    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"091fc9254cd3926924319c8a44fce8cb689d5b08155c1b5ad8d0ba5fb59c0d66\": container with ID starting with 091fc9254cd3926924319c8a44fce8cb689d5b08155c1b5ad8d0ba5fb59c0d66 not found: ID does not exist" containerID="091fc9254cd3926924319c8a44fce8cb689d5b08155c1b5ad8d0ba5fb59c0d66"
Oct 31 01:54:41.950129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.950089    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"091fc9254cd3926924319c8a44fce8cb689d5b08155c1b5ad8d0ba5fb59c0d66"} err="failed to get container status \"091fc9254cd3926924319c8a44fce8cb689d5b08155c1b5ad8d0ba5fb59c0d66\": rpc error: code = NotFound desc = could not find container \"091fc9254cd3926924319c8a44fce8cb689d5b08155c1b5ad8d0ba5fb59c0d66\": container with ID starting with 091fc9254cd3926924319c8a44fce8cb689d5b08155c1b5ad8d0ba5fb59c0d66 not found: ID does not exist"
Oct 31 01:54:41.950129 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.950106    5650 scope.go:122] "RemoveContainer" containerID="9bdc3cbb1a0de66f2b6036dc4ac64920a2853329964824fb0814a1a7348c310d"
Oct 31 01:54:41.961356 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.961336    5650 scope.go:122] "RemoveContainer" containerID="9bdc3cbb1a0de66f2b6036dc4ac64920a2853329964824fb0814a1a7348c310d"
Oct 31 01:54:41.961659 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:54:41.961632    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bdc3cbb1a0de66f2b6036dc4ac64920a2853329964824fb0814a1a7348c310d\": container with ID starting with 9bdc3cbb1a0de66f2b6036dc4ac64920a2853329964824fb0814a1a7348c310d not found: ID does not exist" containerID="9bdc3cbb1a0de66f2b6036dc4ac64920a2853329964824fb0814a1a7348c310d"
Oct 31 01:54:41.961750 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.961669    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bdc3cbb1a0de66f2b6036dc4ac64920a2853329964824fb0814a1a7348c310d"} err="failed to get container status \"9bdc3cbb1a0de66f2b6036dc4ac64920a2853329964824fb0814a1a7348c310d\": rpc error: code = NotFound desc = could not find container \"9bdc3cbb1a0de66f2b6036dc4ac64920a2853329964824fb0814a1a7348c310d\": container with ID starting with 9bdc3cbb1a0de66f2b6036dc4ac64920a2853329964824fb0814a1a7348c310d not found: ID does not exist"
Oct 31 01:54:41.968262 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.968201    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc"]
Oct 31 01:54:41.972174 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.972152    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-726dd1-76dc5b5477-drjkc"]
Oct 31 01:54:41.990736 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.990713    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q"]
Oct 31 01:54:41.995426 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:41.995403    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-726dd1-644c6bb579-hps4q"]
Oct 31 01:54:42.118963 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:42.118932    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c759abc-5d22-4c0a-884e-5d99bcf79fdb" path="/var/lib/kubelet/pods/0c759abc-5d22-4c0a-884e-5d99bcf79fdb/volumes"
Oct 31 01:54:42.119375 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:42.119361    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58ae8229-51c6-4263-a3b2-ec6616c90d0f" path="/var/lib/kubelet/pods/58ae8229-51c6-4263-a3b2-ec6616c90d0f/volumes"
Oct 31 01:54:45.880392 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:45.880349    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"]
Oct 31 01:54:45.880931 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:45.880691    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr" podUID="04d96b2a-ab79-4c0a-a187-766ef1a58c0f" containerName="istio-proxy" containerID="cri-o://bc4c12272e6738a3da9ca1a38aceabd35b8325415641631fda9499f20073ad94" gracePeriod=30
Oct 31 01:54:49.912938 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:49.912908    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh"
Oct 31 01:54:51.023183 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.023153    5650 generic.go:356] "Generic (PLEG): container finished" podID="04d96b2a-ab79-4c0a-a187-766ef1a58c0f" containerID="bc4c12272e6738a3da9ca1a38aceabd35b8325415641631fda9499f20073ad94" exitCode=0
Oct 31 01:54:51.023657 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.023200    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr" event={"ID":"04d96b2a-ab79-4c0a-a187-766ef1a58c0f","Type":"ContainerDied","Data":"bc4c12272e6738a3da9ca1a38aceabd35b8325415641631fda9499f20073ad94"}
Oct 31 01:54:51.023657 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.023247    5650 scope.go:122] "RemoveContainer" containerID="7c31e35ad7c443b677c136f4d46b61312e5fabee20541ea266b9eb982ada8732"
Oct 31 01:54:51.179771 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.179744    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:54:51.274688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.274653    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-workload-socket\") pod \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") "
Oct 31 01:54:51.274877 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.274714    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-token\" (UniqueName: \"kubernetes.io/projected/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-token\") pod \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") "
Oct 31 01:54:51.274925 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.274872    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-envoy\") pod \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") "
Oct 31 01:54:51.274982 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.274963    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-kube-api-access-rr9m9\" (UniqueName: \"kubernetes.io/projected/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-kube-api-access-rr9m9\") pod \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") "
Oct 31 01:54:51.275064 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.275049    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-podinfo\") pod \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") "
Oct 31 01:54:51.275101 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.275085    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istiod-ca-cert\") pod \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") "
Oct 31 01:54:51.275225 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.274962    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-workload-socket" pod "04d96b2a-ab79-4c0a-a187-766ef1a58c0f" (UID: "04d96b2a-ab79-4c0a-a187-766ef1a58c0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:51.275225 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.275116    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-credential-socket\") pod \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") "
Oct 31 01:54:51.275225 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.275177    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-workload-certs\") pod \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") "
Oct 31 01:54:51.275387 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.275231    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-data\") pod \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\" (UID: \"04d96b2a-ab79-4c0a-a187-766ef1a58c0f\") "
Oct 31 01:54:51.275387 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.275376    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-credential-socket" pod "04d96b2a-ab79-4c0a-a187-766ef1a58c0f" (UID: "04d96b2a-ab79-4c0a-a187-766ef1a58c0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:51.275604 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.275565    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-workload-certs" pod "04d96b2a-ab79-4c0a-a187-766ef1a58c0f" (UID: "04d96b2a-ab79-4c0a-a187-766ef1a58c0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:51.275683 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.275659    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-data" pod "04d96b2a-ab79-4c0a-a187-766ef1a58c0f" (UID: "04d96b2a-ab79-4c0a-a187-766ef1a58c0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:51.275797 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.275765    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istiod-ca-cert" pod "04d96b2a-ab79-4c0a-a187-766ef1a58c0f" (UID: "04d96b2a-ab79-4c0a-a187-766ef1a58c0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:51.276283 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.276258    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:51.276383 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.276287    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:51.276383 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.276303    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:51.276383 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.276317    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:51.276383 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.276333    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:51.277284 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.277254    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-podinfo" pod "04d96b2a-ab79-4c0a-a187-766ef1a58c0f" (UID: "04d96b2a-ab79-4c0a-a187-766ef1a58c0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:54:51.277284 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.277265    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-envoy" pod "04d96b2a-ab79-4c0a-a187-766ef1a58c0f" (UID: "04d96b2a-ab79-4c0a-a187-766ef1a58c0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:51.277406 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.277338    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-kube-api-access-rr9m9" pod "04d96b2a-ab79-4c0a-a187-766ef1a58c0f" (UID: "04d96b2a-ab79-4c0a-a187-766ef1a58c0f"). InnerVolumeSpecName "kube-api-access-rr9m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:51.277406 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.277364    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-token" pod "04d96b2a-ab79-4c0a-a187-766ef1a58c0f" (UID: "04d96b2a-ab79-4c0a-a187-766ef1a58c0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:51.378005 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.377967    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:51.378005 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.378001    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:51.378005 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.378015    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:51.378265 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:51.378028    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr9m9\" (UniqueName: \"kubernetes.io/projected/04d96b2a-ab79-4c0a-a187-766ef1a58c0f-kube-api-access-rr9m9\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:54:52.031720 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:52.031690    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"
Oct 31 01:54:52.032168 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:52.031689    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr" event={"ID":"04d96b2a-ab79-4c0a-a187-766ef1a58c0f","Type":"ContainerDied","Data":"176558964bb185aa1b3badd4cda7ba88213ea7f84a1f911fcbbbbb5508c61dcb"}
Oct 31 01:54:52.032168 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:52.031828    5650 scope.go:122] "RemoveContainer" containerID="bc4c12272e6738a3da9ca1a38aceabd35b8325415641631fda9499f20073ad94"
Oct 31 01:54:52.060321 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:52.060286    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"]
Oct 31 01:54:52.067797 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:52.067769    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-726dd1-openshift-default-65bdbcc949-wnczr"]
Oct 31 01:54:52.121197 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:52.121151    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04d96b2a-ab79-4c0a-a187-766ef1a58c0f" path="/var/lib/kubelet/pods/04d96b2a-ab79-4c0a-a187-766ef1a58c0f/volumes"
Oct 31 01:54:54.199472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:54.198721    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:54.310471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:54.309641    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:54.377595 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:54.377543    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:54.491662 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:54.490563    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 01:54:54.698474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:54.694606    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": context deadline exceeded"
Oct 31 01:54:54.826790 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:54.826686    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:54.864087 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:54.864013    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:54.892924 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:54.892881    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 01:54:55.098537 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:55.072887    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded"
Oct 31 01:54:55.100386 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:55.100333    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:55.225084 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:55.225034    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:55.226324 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:55.226285    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 01:54:55.227475 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:55.227433    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:54:55.352620 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:55.351139    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 01:54:55.352620 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:55.351815    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:54:55.415559 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:55.399327    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 01:54:55.415559 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:55.399952    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:54:55.585616 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:55.495425    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:55.585616 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:55.520020    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:54:55.844467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:55.827836    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:55.945470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:55.911556    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:56.019469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:55.965636    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:54:56.019469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:56.016290    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:54:56.072475 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:56.071642    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:56.072682 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:56.072567    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:54:56.121144 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:56.121042    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 31 01:54:56.301860 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:56.261774    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:56.301860 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:56.266487    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 01:54:56.383583 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:56.383475    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:56.406498 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:56.399562    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:56.471525 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:56.466011    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:54:56.471525 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:56.466093    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:54:56.490469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:56.480565    5650 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:54:56.490469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:56.480630    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj" podUID="cd6018a4-4e7e-410a-892f-6b3f800addd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:56.490469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:56.481510    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:54:56.498737 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:56.498690    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 01:54:56.854151 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:56.852664    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded"
Oct 31 01:54:57.034478 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:57.026540    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:57.076710 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:57.072966    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 01:54:57.087579 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:57.087544    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:54:57.096622 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:57.096594    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:54:57.170619 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:57.169220    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:54:57.171278 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:57.171247    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:54:57.186640 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:57.186549    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:54:57.187237 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:57.187217    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:54:57.194139 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:57.193146    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:54:57.363709 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:54:57.363674    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:54:58.577897 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:54:58.577862    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:55:03.502549 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.502506    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n"]
Oct 31 01:55:03.503532 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.503508    5650 cpu_manager.go:414] "containerMap: removing container" podUID="58ae8229-51c6-4263-a3b2-ec6616c90d0f" containerName="payload-processing"
Oct 31 01:55:03.503631 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.503533    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58ae8229-51c6-4263-a3b2-ec6616c90d0f" containerName="payload-processing"
Oct 31 01:55:03.503631 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.503553    5650 cpu_manager.go:414] "containerMap: removing container" podUID="0c759abc-5d22-4c0a-884e-5d99bcf79fdb" containerName="payload-pre-processing"
Oct 31 01:55:03.503631 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.503562    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c759abc-5d22-4c0a-884e-5d99bcf79fdb" containerName="payload-pre-processing"
Oct 31 01:55:03.503631 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.503605    5650 cpu_manager.go:414] "containerMap: removing container" podUID="04d96b2a-ab79-4c0a-a187-766ef1a58c0f" containerName="istio-proxy"
Oct 31 01:55:03.503631 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.503613    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04d96b2a-ab79-4c0a-a187-766ef1a58c0f" containerName="istio-proxy"
Oct 31 01:55:03.503794 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.503643    5650 cpu_manager.go:414] "containerMap: removing container" podUID="04d96b2a-ab79-4c0a-a187-766ef1a58c0f" containerName="istio-proxy"
Oct 31 01:55:03.503794 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.503651    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04d96b2a-ab79-4c0a-a187-766ef1a58c0f" containerName="istio-proxy"
Oct 31 01:55:03.503861 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.503810    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="58ae8229-51c6-4263-a3b2-ec6616c90d0f" containerName="payload-processing"
Oct 31 01:55:03.503861 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.503822    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c759abc-5d22-4c0a-884e-5d99bcf79fdb" containerName="payload-pre-processing"
Oct 31 01:55:03.503861 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.503838    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="04d96b2a-ab79-4c0a-a187-766ef1a58c0f" containerName="istio-proxy"
Oct 31 01:55:03.503861 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.503851    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="04d96b2a-ab79-4c0a-a187-766ef1a58c0f" containerName="istio-proxy"
Oct 31 01:55:03.542104 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.540437    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n"]
Oct 31 01:55:03.542104 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.540541    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n"
Oct 31 01:55:03.547255 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.547229    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:55:03.547356 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.547275    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:55:03.580364 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.580322    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55644f09-e9ef-415f-8cb7-9ccbc7b400b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n\" (UID: \"55644f09-e9ef-415f-8cb7-9ccbc7b400b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n"
Oct 31 01:55:03.580661 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.580399    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pbdp\" (UniqueName: \"kubernetes.io/projected/55644f09-e9ef-415f-8cb7-9ccbc7b400b5-kube-api-access-8pbdp\") pod \"maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n\" (UID: \"55644f09-e9ef-415f-8cb7-9ccbc7b400b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n"
Oct 31 01:55:03.580661 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.580505    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55644f09-e9ef-415f-8cb7-9ccbc7b400b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n\" (UID: \"55644f09-e9ef-415f-8cb7-9ccbc7b400b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n"
Oct 31 01:55:03.681680 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.681635    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55644f09-e9ef-415f-8cb7-9ccbc7b400b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n\" (UID: \"55644f09-e9ef-415f-8cb7-9ccbc7b400b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n"
Oct 31 01:55:03.681878 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.681693    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pbdp\" (UniqueName: \"kubernetes.io/projected/55644f09-e9ef-415f-8cb7-9ccbc7b400b5-kube-api-access-8pbdp\") pod \"maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n\" (UID: \"55644f09-e9ef-415f-8cb7-9ccbc7b400b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n"
Oct 31 01:55:03.681878 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.681757    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55644f09-e9ef-415f-8cb7-9ccbc7b400b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n\" (UID: \"55644f09-e9ef-415f-8cb7-9ccbc7b400b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n"
Oct 31 01:55:03.682353 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.682325    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55644f09-e9ef-415f-8cb7-9ccbc7b400b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n\" (UID: \"55644f09-e9ef-415f-8cb7-9ccbc7b400b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n"
Oct 31 01:55:03.684174 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.684152    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55644f09-e9ef-415f-8cb7-9ccbc7b400b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n\" (UID: \"55644f09-e9ef-415f-8cb7-9ccbc7b400b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n"
Oct 31 01:55:03.695425 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.695393    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pbdp\" (UniqueName: \"kubernetes.io/projected/55644f09-e9ef-415f-8cb7-9ccbc7b400b5-kube-api-access-8pbdp\") pod \"maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n\" (UID: \"55644f09-e9ef-415f-8cb7-9ccbc7b400b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n"
Oct 31 01:55:03.893346 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:03.893269    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n"
Oct 31 01:55:04.242867 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:04.242836    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n"]
Oct 31 01:55:04.245606 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:55:04.245567    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55644f09_e9ef_415f_8cb7_9ccbc7b400b5.slice/crio-8876e0f6b8533e9711f78505d292e3e216f6ea5b78c726b867270798b53b88cf WatchSource:0}: Error finding container 8876e0f6b8533e9711f78505d292e3e216f6ea5b78c726b867270798b53b88cf: Status 404 returned error can't find the container with id 8876e0f6b8533e9711f78505d292e3e216f6ea5b78c726b867270798b53b88cf
Oct 31 01:55:04.949343 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:04.949247    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n" event={"ID":"55644f09-e9ef-415f-8cb7-9ccbc7b400b5","Type":"ContainerStarted","Data":"db62156159d40902d78d24677dcea511d5d4aa504f4dcaaf60f89b8995029ecf"}
Oct 31 01:55:04.949343 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:04.949303    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n" event={"ID":"55644f09-e9ef-415f-8cb7-9ccbc7b400b5","Type":"ContainerStarted","Data":"8876e0f6b8533e9711f78505d292e3e216f6ea5b78c726b867270798b53b88cf"}
Oct 31 01:55:04.999756 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:04.999702    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n" podStartSLOduration=1.999687952 podStartE2EDuration="1.999687952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:04.996420376 +0000 UTC m=+2061.000134679" watchObservedRunningTime="2026-10-31 01:55:04.999687952 +0000 UTC m=+2061.003402243"
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.547420    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh"]
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.572910    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh"]
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.572961    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk"]
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.573026    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh"
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.582528    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-ed4dce\""
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.607246    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk"]
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.607374    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk"
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.613125    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-ed4dce-dockercfg-ggp7x\""
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.616699    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a3c58e-ec8b-4dd8-94dd-9c4674840b30-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk\" (UID: \"f4a3c58e-ec8b-4dd8-94dd-9c4674840b30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk"
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.616796    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fppc6\" (UniqueName: \"kubernetes.io/projected/877fff39-e2ad-488e-a14a-60d71bbe02f5-kube-api-access-fppc6\") pod \"payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh\" (UID: \"877fff39-e2ad-488e-a14a-60d71bbe02f5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh"
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.616920    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbh2v\" (UniqueName: \"kubernetes.io/projected/f4a3c58e-ec8b-4dd8-94dd-9c4674840b30-kube-api-access-wbh2v\") pod \"payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk\" (UID: \"f4a3c58e-ec8b-4dd8-94dd-9c4674840b30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk"
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.616964    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/877fff39-e2ad-488e-a14a-60d71bbe02f5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh\" (UID: \"877fff39-e2ad-488e-a14a-60d71bbe02f5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh"
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.718005    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a3c58e-ec8b-4dd8-94dd-9c4674840b30-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk\" (UID: \"f4a3c58e-ec8b-4dd8-94dd-9c4674840b30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk"
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.718188    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fppc6\" (UniqueName: \"kubernetes.io/projected/877fff39-e2ad-488e-a14a-60d71bbe02f5-kube-api-access-fppc6\") pod \"payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh\" (UID: \"877fff39-e2ad-488e-a14a-60d71bbe02f5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh"
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.718332    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbh2v\" (UniqueName: \"kubernetes.io/projected/f4a3c58e-ec8b-4dd8-94dd-9c4674840b30-kube-api-access-wbh2v\") pod \"payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk\" (UID: \"f4a3c58e-ec8b-4dd8-94dd-9c4674840b30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk"
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.718374    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/877fff39-e2ad-488e-a14a-60d71bbe02f5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh\" (UID: \"877fff39-e2ad-488e-a14a-60d71bbe02f5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh"
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.718735    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a3c58e-ec8b-4dd8-94dd-9c4674840b30-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk\" (UID: \"f4a3c58e-ec8b-4dd8-94dd-9c4674840b30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk"
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.719236    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/877fff39-e2ad-488e-a14a-60d71bbe02f5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh\" (UID: \"877fff39-e2ad-488e-a14a-60d71bbe02f5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh"
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.732103    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbh2v\" (UniqueName: \"kubernetes.io/projected/f4a3c58e-ec8b-4dd8-94dd-9c4674840b30-kube-api-access-wbh2v\") pod \"payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk\" (UID: \"f4a3c58e-ec8b-4dd8-94dd-9c4674840b30\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk"
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.732117    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fppc6\" (UniqueName: \"kubernetes.io/projected/877fff39-e2ad-488e-a14a-60d71bbe02f5-kube-api-access-fppc6\") pod \"payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh\" (UID: \"877fff39-e2ad-488e-a14a-60d71bbe02f5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh"
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.909644    5650 patch_prober.go:29] interesting pod/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.909711    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z" podUID="f6011af8-136f-436a-bb13-987d5844f336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.925618    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh"
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.947965    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk"
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.962402    5650 generic.go:356] "Generic (PLEG): container finished" podID="55644f09-e9ef-415f-8cb7-9ccbc7b400b5" containerID="db62156159d40902d78d24677dcea511d5d4aa504f4dcaaf60f89b8995029ecf" exitCode=0
Oct 31 01:55:06.119880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:05.962583    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n" event={"ID":"55644f09-e9ef-415f-8cb7-9ccbc7b400b5","Type":"ContainerDied","Data":"db62156159d40902d78d24677dcea511d5d4aa504f4dcaaf60f89b8995029ecf"}
Oct 31 01:55:06.317916 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:06.249710    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh"]
Oct 31 01:55:06.317916 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:55:06.252584    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877fff39_e2ad_488e_a14a_60d71bbe02f5.slice/crio-aa76147e7b087d5ca6f2b5b2f1d82d3cd9d76025bb47905dc854b6a9d5c1d075 WatchSource:0}: Error finding container aa76147e7b087d5ca6f2b5b2f1d82d3cd9d76025bb47905dc854b6a9d5c1d075: Status 404 returned error can't find the container with id aa76147e7b087d5ca6f2b5b2f1d82d3cd9d76025bb47905dc854b6a9d5c1d075
Oct 31 01:55:06.317916 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:06.287195    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk"]
Oct 31 01:55:06.317916 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:55:06.288569    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a3c58e_ec8b_4dd8_94dd_9c4674840b30.slice/crio-0deda906384285b7b95bc4c3645a4efb7595b9a12b1b7e7b6f4450c662425193 WatchSource:0}: Error finding container 0deda906384285b7b95bc4c3645a4efb7595b9a12b1b7e7b6f4450c662425193: Status 404 returned error can't find the container with id 0deda906384285b7b95bc4c3645a4efb7595b9a12b1b7e7b6f4450c662425193
Oct 31 01:55:06.971891 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:06.971859    5650 generic.go:356] "Generic (PLEG): container finished" podID="f6011af8-136f-436a-bb13-987d5844f336" containerID="bbba164712e42fa94a9c7cc2065cc488af9e37267c86017d4c457dfc8af744fa" exitCode=0
Oct 31 01:55:06.972077 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:06.971935    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z" event={"ID":"f6011af8-136f-436a-bb13-987d5844f336","Type":"ContainerDied","Data":"bbba164712e42fa94a9c7cc2065cc488af9e37267c86017d4c457dfc8af744fa"}
Oct 31 01:55:06.972497 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:06.972473    5650 scope.go:122] "RemoveContainer" containerID="bbba164712e42fa94a9c7cc2065cc488af9e37267c86017d4c457dfc8af744fa"
Oct 31 01:55:06.973794 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:06.973767    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk" event={"ID":"f4a3c58e-ec8b-4dd8-94dd-9c4674840b30","Type":"ContainerStarted","Data":"a47e9d3338d3aa0541aaa177b65c5da550c7c1fa2b68d3f2b1b4b5562cfad2a6"}
Oct 31 01:55:06.973921 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:06.973824    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk"
Oct 31 01:55:06.973921 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:06.973849    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk" event={"ID":"f4a3c58e-ec8b-4dd8-94dd-9c4674840b30","Type":"ContainerStarted","Data":"0deda906384285b7b95bc4c3645a4efb7595b9a12b1b7e7b6f4450c662425193"}
Oct 31 01:55:06.974128 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:06.974099    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:55:06.974199 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:06.974183    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:55:06.974251 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:06.974229    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:55:06.975505 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:06.975480    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh" event={"ID":"877fff39-e2ad-488e-a14a-60d71bbe02f5","Type":"ContainerStarted","Data":"bc30f0ee9541c90b14e8874827379571f19c10db7cae9ea00b86eb861917c3b5"}
Oct 31 01:55:06.975617 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:06.975528    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh" event={"ID":"877fff39-e2ad-488e-a14a-60d71bbe02f5","Type":"ContainerStarted","Data":"aa76147e7b087d5ca6f2b5b2f1d82d3cd9d76025bb47905dc854b6a9d5c1d075"}
Oct 31 01:55:06.975617 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:06.975553    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh"
Oct 31 01:55:06.977265 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:06.977239    5650 generic.go:356] "Generic (PLEG): container finished" podID="7fe243de-3376-4d8d-87bd-032e29f0083d" containerID="f7de8e8e314a30de871cf03eccd5bf5a7fd6b496e395ebc93be647095e74431a" exitCode=0
Oct 31 01:55:06.977363 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:06.977301    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w" event={"ID":"7fe243de-3376-4d8d-87bd-032e29f0083d","Type":"ContainerDied","Data":"f7de8e8e314a30de871cf03eccd5bf5a7fd6b496e395ebc93be647095e74431a"}
Oct 31 01:55:06.977771 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:06.977754    5650 scope.go:122] "RemoveContainer" containerID="f7de8e8e314a30de871cf03eccd5bf5a7fd6b496e395ebc93be647095e74431a"
Oct 31 01:55:06.987611 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:06.987436    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:55:06.987611 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:06.987541    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:55:06.987611 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:06.987583    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:55:07.022389 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:07.022326    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk" podStartSLOduration=2.022306479 podStartE2EDuration="2.022306479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:07.020894375 +0000 UTC m=+2063.024608668" watchObservedRunningTime="2026-10-31 01:55:07.022306479 +0000 UTC m=+2063.026020771"
Oct 31 01:55:07.089637 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:07.089498    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh" podStartSLOduration=2.089478003 podStartE2EDuration="2.089478003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:07.085905976 +0000 UTC m=+2063.089620267" watchObservedRunningTime="2026-10-31 01:55:07.089478003 +0000 UTC m=+2063.093192293"
Oct 31 01:55:07.244251 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:07.244216    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n"
Oct 31 01:55:07.462237 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:07.462185    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55644f09-e9ef-415f-8cb7-9ccbc7b400b5-kube-api-access-8pbdp\" (UniqueName: \"kubernetes.io/projected/55644f09-e9ef-415f-8cb7-9ccbc7b400b5-kube-api-access-8pbdp\") pod \"55644f09-e9ef-415f-8cb7-9ccbc7b400b5\" (UID: \"55644f09-e9ef-415f-8cb7-9ccbc7b400b5\") "
Oct 31 01:55:07.462427 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:07.462380    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55644f09-e9ef-415f-8cb7-9ccbc7b400b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55644f09-e9ef-415f-8cb7-9ccbc7b400b5-maas-api-openshift-ca\") pod \"55644f09-e9ef-415f-8cb7-9ccbc7b400b5\" (UID: \"55644f09-e9ef-415f-8cb7-9ccbc7b400b5\") "
Oct 31 01:55:07.462579 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:07.462477    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55644f09-e9ef-415f-8cb7-9ccbc7b400b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55644f09-e9ef-415f-8cb7-9ccbc7b400b5-maas-api-xks-ca\") pod \"55644f09-e9ef-415f-8cb7-9ccbc7b400b5\" (UID: \"55644f09-e9ef-415f-8cb7-9ccbc7b400b5\") "
Oct 31 01:55:07.462838 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:07.462809    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55644f09-e9ef-415f-8cb7-9ccbc7b400b5-maas-api-openshift-ca" pod "55644f09-e9ef-415f-8cb7-9ccbc7b400b5" (UID: "55644f09-e9ef-415f-8cb7-9ccbc7b400b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:55:07.463117 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:07.463098    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55644f09-e9ef-415f-8cb7-9ccbc7b400b5-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:07.464700 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:07.464669    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55644f09-e9ef-415f-8cb7-9ccbc7b400b5-maas-api-xks-ca" pod "55644f09-e9ef-415f-8cb7-9ccbc7b400b5" (UID: "55644f09-e9ef-415f-8cb7-9ccbc7b400b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:55:07.465061 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:07.465023    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55644f09-e9ef-415f-8cb7-9ccbc7b400b5-kube-api-access-8pbdp" pod "55644f09-e9ef-415f-8cb7-9ccbc7b400b5" (UID: "55644f09-e9ef-415f-8cb7-9ccbc7b400b5"). InnerVolumeSpecName "kube-api-access-8pbdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:55:07.564530 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:07.564493    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55644f09-e9ef-415f-8cb7-9ccbc7b400b5-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:07.564530 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:07.564528    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pbdp\" (UniqueName: \"kubernetes.io/projected/55644f09-e9ef-415f-8cb7-9ccbc7b400b5-kube-api-access-8pbdp\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:07.989514 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:07.989469    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w" event={"ID":"7fe243de-3376-4d8d-87bd-032e29f0083d","Type":"ContainerStarted","Data":"87fce5c7e3a690205e8e1479f0bcf075a4a07c7e91f4a2f34c89863903a5fdc5"}
Oct 31 01:55:07.993412 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:07.993026    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z" event={"ID":"f6011af8-136f-436a-bb13-987d5844f336","Type":"ContainerStarted","Data":"5308604b5f247586a17fad9cb72a29828c5cab523852a19adae7b777b07f7f6a"}
Oct 31 01:55:08.012106 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:08.012013    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n"
Oct 31 01:55:08.029652 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:08.029609    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n" event={"ID":"55644f09-e9ef-415f-8cb7-9ccbc7b400b5","Type":"ContainerDied","Data":"8876e0f6b8533e9711f78505d292e3e216f6ea5b78c726b867270798b53b88cf"}
Oct 31 01:55:08.029912 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:08.029890    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8876e0f6b8533e9711f78505d292e3e216f6ea5b78c726b867270798b53b88cf"
Oct 31 01:55:08.820470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:08.817963    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:55:08.820470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:08.818111    5650 patch_prober.go:29] interesting pod/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 01:55:08.820470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:08.818157    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z" podUID="f6011af8-136f-436a-bb13-987d5844f336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 01:55:08.849739 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:08.849546    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:55:08.850342 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:08.850310    5650 patch_prober.go:29] interesting pod/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 31 01:55:08.850519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:08.850368    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w" podUID="7fe243de-3376-4d8d-87bd-032e29f0083d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 31 01:55:09.809473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:09.807565    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:55:09.826471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:09.824878    5650 patch_prober.go:29] interesting pod/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 01:55:09.826471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:09.824944    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z" podUID="f6011af8-136f-436a-bb13-987d5844f336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 01:55:09.849471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:09.849027    5650 patch_prober.go:29] interesting pod/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 31 01:55:09.849471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:09.849075    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w" podUID="7fe243de-3376-4d8d-87bd-032e29f0083d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 31 01:55:10.819474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:10.817935    5650 patch_prober.go:29] interesting pod/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 01:55:10.819474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:10.818006    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z" podUID="f6011af8-136f-436a-bb13-987d5844f336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 01:55:10.848473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:10.848091    5650 patch_prober.go:29] interesting pod/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 31 01:55:10.848473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:10.848157    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w" podUID="7fe243de-3376-4d8d-87bd-032e29f0083d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 31 01:55:11.815472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:11.813941    5650 patch_prober.go:29] interesting pod/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 01:55:11.815472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:11.814026    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z" podUID="f6011af8-136f-436a-bb13-987d5844f336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 01:55:11.855049 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:11.855002    5650 patch_prober.go:29] interesting pod/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 31 01:55:11.855740 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:11.855703    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w" podUID="7fe243de-3376-4d8d-87bd-032e29f0083d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 31 01:55:12.816918 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:12.816875    5650 patch_prober.go:29] interesting pod/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 01:55:12.817274 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:12.817241    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z" podUID="f6011af8-136f-436a-bb13-987d5844f336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 01:55:12.854465 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:12.854046    5650 patch_prober.go:29] interesting pod/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 31 01:55:12.854465 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:12.854141    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w" podUID="7fe243de-3376-4d8d-87bd-032e29f0083d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 31 01:55:13.653852 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:13.653811    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv"]
Oct 31 01:55:13.665017 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:13.664983    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4"]
Oct 31 01:55:13.817889 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:13.817855    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:55:13.818263 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:13.818238    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:55:13.818976 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:13.818955    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:55:13.853567 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:13.853537    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:55:13.854556 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:13.854533    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:55:14.801722 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:14.801685    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n"]
Oct 31 01:55:14.806588 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:14.806543    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-eb8ae6-crt5n"]
Oct 31 01:55:16.208504 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:16.205106    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55644f09-e9ef-415f-8cb7-9ccbc7b400b5" path="/var/lib/kubelet/pods/55644f09-e9ef-415f-8cb7-9ccbc7b400b5/volumes"
Oct 31 01:55:17.393115 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:17.218563    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 01:55:17.393115 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:17.283132    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:17.393725 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:17.393686    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:17.473420 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:17.473379    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:17.500486 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:17.495669    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:17.500486 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:17.499565    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:18.071158 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:18.068651    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh"
Oct 31 01:55:18.102472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:18.088707    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk"
Oct 31 01:55:18.102472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:18.089792    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:18.129929 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:18.128615    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 01:55:18.187072 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:18.187027    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 01:55:18.274092 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:18.274045    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 01:55:18.296646 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:18.296599    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 01:55:18.344532 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:18.311146    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded"
Oct 31 01:55:18.432522 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:18.405029    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 01:55:18.501472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:18.470568    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:55:18.509471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:18.505605    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:55:18.509806 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:18.509778    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:55:18.532259 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:18.532189    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:18.533540 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:18.532938    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:18.533540 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:18.533303    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:55:18.533540 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:18.533360    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:55:18.603624 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:18.603537    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 01:55:18.671024 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:18.670988    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:55:18.671461 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:18.671418    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv" podUID="9feea519-15e9-4a7b-bd63-ab6bf4965707" containerName="payload-processing" containerID="cri-o://fbf9e78e5de5a28e6dd32eda8ae2d7b7f1402036e98a0532b554fc3d3d9f349b" gracePeriod=25
Oct 31 01:55:19.159472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:18.937311    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4" podUID="fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91" containerName="payload-pre-processing" containerID="cri-o://0a1779eed9d1a414ae465c5c00a4021e216bab054fe9b11fe4edb2ac13222483" gracePeriod=25
Oct 31 01:55:19.301936 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.159802    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:19.302195 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.159880    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 31 01:55:19.302306 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.163380    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded"
Oct 31 01:55:19.361897 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.361864    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:55:19.362188 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.362166    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:55:19.370377 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.364033    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 01:55:19.370377 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.364113    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 01:55:19.370377 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.364575    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:19.419568 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.417226    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:55:19.440483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.438812    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 01:55:19.463688 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.463657    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 01:55:19.514785 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.506716    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:19.576364 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.575401    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Oct 31 01:55:19.576364 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.575840    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 01:55:19.611737 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.611709    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 01:55:19.613484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.613407    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 01:55:19.624064 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.624035    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 01:55:19.697751 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.697643    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:55:19.699468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.699411    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"]
Oct 31 01:55:19.700627 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.700589    5650 cpu_manager.go:414] "containerMap: removing container" podUID="55644f09-e9ef-415f-8cb7-9ccbc7b400b5" containerName="revoke-keys"
Oct 31 01:55:19.700814 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.700791    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55644f09-e9ef-415f-8cb7-9ccbc7b400b5" containerName="revoke-keys"
Oct 31 01:55:19.701105 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.701081    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="55644f09-e9ef-415f-8cb7-9ccbc7b400b5" containerName="revoke-keys"
Oct 31 01:55:19.708686 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.705521    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.712496 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.710152    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:55:19.715258 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.714963    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-46a67e-openshift-default-dockercfg-ttzhz\""
Oct 31 01:55:19.745855 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.744868    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"]
Oct 31 01:55:19.746489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.746236    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 01:55:19.752369 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.752344    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 01:55:19.814645 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.812810    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-workload-socket\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.814645 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.812878    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d2pd\" (UniqueName: \"kubernetes.io/projected/fe7e7396-a227-4486-97ba-8b48945cc1d3-kube-api-access-5d2pd\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.814645 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.812940    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe7e7396-a227-4486-97ba-8b48945cc1d3-istiod-ca-cert\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.814645 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.812995    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-data\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.814645 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.813024    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-envoy\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.814645 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.813082    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-credential-socket\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.814645 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.813197    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-podinfo\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.814645 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.813410    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-workload-certs\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.814645 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.813457    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-token\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.914902 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.914851    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-workload-socket\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.915158 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.914934    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d2pd\" (UniqueName: \"kubernetes.io/projected/fe7e7396-a227-4486-97ba-8b48945cc1d3-kube-api-access-5d2pd\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.915158 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.915001    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe7e7396-a227-4486-97ba-8b48945cc1d3-istiod-ca-cert\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.915158 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.915065    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-data\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.915158 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.915098    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-envoy\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.915158 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.915145    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-credential-socket\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.915483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.915248    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-podinfo\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.915483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.915363    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-workload-socket\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.915483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.915417    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-workload-certs\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.915483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.915461    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-token\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.916122 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.916096    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-data\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.916544 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.916127    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-credential-socket\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.916544 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.916146    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe7e7396-a227-4486-97ba-8b48945cc1d3-istiod-ca-cert\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.916544 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.916362    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-workload-certs\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.918672 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.918646    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-envoy\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.918829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.918813    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-podinfo\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.936359 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.936329    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d2pd\" (UniqueName: \"kubernetes.io/projected/fe7e7396-a227-4486-97ba-8b48945cc1d3-kube-api-access-5d2pd\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.941557 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.939436    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-token\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:19.970153 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.970122    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv"
Oct 31 01:55:19.973618 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:19.973588    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4"
Oct 31 01:55:20.051140 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.051061    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:20.119387 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.118503    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9feea519-15e9-4a7b-bd63-ab6bf4965707-kube-api-access-vlk98\" (UniqueName: \"kubernetes.io/projected/9feea519-15e9-4a7b-bd63-ab6bf4965707-kube-api-access-vlk98\") pod \"9feea519-15e9-4a7b-bd63-ab6bf4965707\" (UID: \"9feea519-15e9-4a7b-bd63-ab6bf4965707\") "
Oct 31 01:55:20.119387 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.118574    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91-plugins-config-volume\") pod \"fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91\" (UID: \"fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91\") "
Oct 31 01:55:20.119387 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.118805    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91-kube-api-access-hbnng\" (UniqueName: \"kubernetes.io/projected/fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91-kube-api-access-hbnng\") pod \"fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91\" (UID: \"fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91\") "
Oct 31 01:55:20.119387 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.118851    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9feea519-15e9-4a7b-bd63-ab6bf4965707-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9feea519-15e9-4a7b-bd63-ab6bf4965707-plugins-config-volume\") pod \"9feea519-15e9-4a7b-bd63-ab6bf4965707\" (UID: \"9feea519-15e9-4a7b-bd63-ab6bf4965707\") "
Oct 31 01:55:20.119800 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.119782    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9feea519-15e9-4a7b-bd63-ab6bf4965707-plugins-config-volume" pod "9feea519-15e9-4a7b-bd63-ab6bf4965707" (UID: "9feea519-15e9-4a7b-bd63-ab6bf4965707"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:55:20.120567 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.120545    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91-plugins-config-volume" pod "fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91" (UID: "fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:55:20.126257 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.126230    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9feea519-15e9-4a7b-bd63-ab6bf4965707-kube-api-access-vlk98" pod "9feea519-15e9-4a7b-bd63-ab6bf4965707" (UID: "9feea519-15e9-4a7b-bd63-ab6bf4965707"). InnerVolumeSpecName "kube-api-access-vlk98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:55:20.126957 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.126928    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91-kube-api-access-hbnng" pod "fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91" (UID: "fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91"). InnerVolumeSpecName "kube-api-access-hbnng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:55:20.221537 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.221406    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:20.221537 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.221462    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbnng\" (UniqueName: \"kubernetes.io/projected/fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91-kube-api-access-hbnng\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:20.221537 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.221479    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9feea519-15e9-4a7b-bd63-ab6bf4965707-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:20.221537 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.221495    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlk98\" (UniqueName: \"kubernetes.io/projected/9feea519-15e9-4a7b-bd63-ab6bf4965707-kube-api-access-vlk98\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:20.366372 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:55:20.366129    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe7e7396_a227_4486_97ba_8b48945cc1d3.slice/crio-cafa78e9ce3e140ddcf83c6c0a7c2aacc57f7d9983cb84ef8bf84934adfaf6de WatchSource:0}: Error finding container cafa78e9ce3e140ddcf83c6c0a7c2aacc57f7d9983cb84ef8bf84934adfaf6de: Status 404 returned error can't find the container with id cafa78e9ce3e140ddcf83c6c0a7c2aacc57f7d9983cb84ef8bf84934adfaf6de
Oct 31 01:55:20.367555 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.367404    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"]
Oct 31 01:55:20.369981 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.369942    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:55:20.370070 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.370030    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:55:20.370112 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.370070    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:55:20.386155 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.386118    5650 generic.go:356] "Generic (PLEG): container finished" podID="9feea519-15e9-4a7b-bd63-ab6bf4965707" containerID="fbf9e78e5de5a28e6dd32eda8ae2d7b7f1402036e98a0532b554fc3d3d9f349b" exitCode=0
Oct 31 01:55:20.386375 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.386182    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv"
Oct 31 01:55:20.386375 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.386207    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv" event={"ID":"9feea519-15e9-4a7b-bd63-ab6bf4965707","Type":"ContainerDied","Data":"fbf9e78e5de5a28e6dd32eda8ae2d7b7f1402036e98a0532b554fc3d3d9f349b"}
Oct 31 01:55:20.386375 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.386280    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv" event={"ID":"9feea519-15e9-4a7b-bd63-ab6bf4965707","Type":"ContainerDied","Data":"651f359ff16384c74ac72a0fff4c3fd42e18673ffcee49aebd669476e224942d"}
Oct 31 01:55:20.386375 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.386314    5650 scope.go:122] "RemoveContainer" containerID="fbf9e78e5de5a28e6dd32eda8ae2d7b7f1402036e98a0532b554fc3d3d9f349b"
Oct 31 01:55:20.402041 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.402005    5650 generic.go:356] "Generic (PLEG): container finished" podID="fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91" containerID="0a1779eed9d1a414ae465c5c00a4021e216bab054fe9b11fe4edb2ac13222483" exitCode=0
Oct 31 01:55:20.402182 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.402122    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4" event={"ID":"fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91","Type":"ContainerDied","Data":"0a1779eed9d1a414ae465c5c00a4021e216bab054fe9b11fe4edb2ac13222483"}
Oct 31 01:55:20.402182 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.402168    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4" event={"ID":"fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91","Type":"ContainerDied","Data":"6c56969a5a8cb2853ac42806f41f7d29491926371a9332ca35c76ca362eca307"}
Oct 31 01:55:20.402262 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.402235    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4"
Oct 31 01:55:20.418474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.417488    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv"]
Oct 31 01:55:20.422986 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.422726    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8" event={"ID":"fe7e7396-a227-4486-97ba-8b48945cc1d3","Type":"ContainerStarted","Data":"cafa78e9ce3e140ddcf83c6c0a7c2aacc57f7d9983cb84ef8bf84934adfaf6de"}
Oct 31 01:55:20.445222 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.445185    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-eb8ae6-7789f8f95c-g4knv"]
Oct 31 01:55:20.446594 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.446570    5650 scope.go:122] "RemoveContainer" containerID="fbf9e78e5de5a28e6dd32eda8ae2d7b7f1402036e98a0532b554fc3d3d9f349b"
Oct 31 01:55:20.446991 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:55:20.446959    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbf9e78e5de5a28e6dd32eda8ae2d7b7f1402036e98a0532b554fc3d3d9f349b\": container with ID starting with fbf9e78e5de5a28e6dd32eda8ae2d7b7f1402036e98a0532b554fc3d3d9f349b not found: ID does not exist" containerID="fbf9e78e5de5a28e6dd32eda8ae2d7b7f1402036e98a0532b554fc3d3d9f349b"
Oct 31 01:55:20.447087 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.447002    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbf9e78e5de5a28e6dd32eda8ae2d7b7f1402036e98a0532b554fc3d3d9f349b"} err="failed to get container status \"fbf9e78e5de5a28e6dd32eda8ae2d7b7f1402036e98a0532b554fc3d3d9f349b\": rpc error: code = NotFound desc = could not find container \"fbf9e78e5de5a28e6dd32eda8ae2d7b7f1402036e98a0532b554fc3d3d9f349b\": container with ID starting with fbf9e78e5de5a28e6dd32eda8ae2d7b7f1402036e98a0532b554fc3d3d9f349b not found: ID does not exist"
Oct 31 01:55:20.447087 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.447020    5650 scope.go:122] "RemoveContainer" containerID="0a1779eed9d1a414ae465c5c00a4021e216bab054fe9b11fe4edb2ac13222483"
Oct 31 01:55:20.468537 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.468432    5650 scope.go:122] "RemoveContainer" containerID="0a1779eed9d1a414ae465c5c00a4021e216bab054fe9b11fe4edb2ac13222483"
Oct 31 01:55:20.469352 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:55:20.469324    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a1779eed9d1a414ae465c5c00a4021e216bab054fe9b11fe4edb2ac13222483\": container with ID starting with 0a1779eed9d1a414ae465c5c00a4021e216bab054fe9b11fe4edb2ac13222483 not found: ID does not exist" containerID="0a1779eed9d1a414ae465c5c00a4021e216bab054fe9b11fe4edb2ac13222483"
Oct 31 01:55:20.469469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.469363    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a1779eed9d1a414ae465c5c00a4021e216bab054fe9b11fe4edb2ac13222483"} err="failed to get container status \"0a1779eed9d1a414ae465c5c00a4021e216bab054fe9b11fe4edb2ac13222483\": rpc error: code = NotFound desc = could not find container \"0a1779eed9d1a414ae465c5c00a4021e216bab054fe9b11fe4edb2ac13222483\": container with ID starting with 0a1779eed9d1a414ae465c5c00a4021e216bab054fe9b11fe4edb2ac13222483 not found: ID does not exist"
Oct 31 01:55:20.484320 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.484292    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4"]
Oct 31 01:55:20.490353 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:20.489163    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eb8ae6-cf677f7ff-d4rl4"]
Oct 31 01:55:21.433924 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:21.433883    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8" event={"ID":"fe7e7396-a227-4486-97ba-8b48945cc1d3","Type":"ContainerStarted","Data":"273ebfeeb08e9008b5e71fc27415a534e938057c3de950849f7161c7bc6f14d8"}
Oct 31 01:55:21.476146 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:21.476093    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8" podStartSLOduration=2.476059872 podStartE2EDuration="2.476059872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:21.47430937 +0000 UTC m=+2077.478023658" watchObservedRunningTime="2026-10-31 01:55:21.476059872 +0000 UTC m=+2077.479774162"
Oct 31 01:55:22.052211 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:22.052172    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:22.057829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:22.057802    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:22.119317 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:22.119281    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9feea519-15e9-4a7b-bd63-ab6bf4965707" path="/var/lib/kubelet/pods/9feea519-15e9-4a7b-bd63-ab6bf4965707/volumes"
Oct 31 01:55:22.119805 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:22.119790    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91" path="/var/lib/kubelet/pods/fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91/volumes"
Oct 31 01:55:22.444833 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:22.444744    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:22.451091 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:22.451041    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:55:23.387911 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:23.387877    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"]
Oct 31 01:55:23.388531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:23.388080    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w" podUID="7fe243de-3376-4d8d-87bd-032e29f0083d" containerName="istio-proxy" containerID="cri-o://87fce5c7e3a690205e8e1479f0bcf075a4a07c7e91f4a2f34c89863903a5fdc5" gracePeriod=30
Oct 31 01:55:25.599326 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:25.598277    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddba8822-rcwh7"]
Oct 31 01:55:25.599326 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:25.598957    5650 cpu_manager.go:414] "containerMap: removing container" podUID="fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91" containerName="payload-pre-processing"
Oct 31 01:55:25.599326 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:25.598972    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91" containerName="payload-pre-processing"
Oct 31 01:55:25.599326 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:25.599006    5650 cpu_manager.go:414] "containerMap: removing container" podUID="9feea519-15e9-4a7b-bd63-ab6bf4965707" containerName="payload-processing"
Oct 31 01:55:25.599326 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:25.599011    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9feea519-15e9-4a7b-bd63-ab6bf4965707" containerName="payload-processing"
Oct 31 01:55:25.599326 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:25.599115    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb5bf02e-6f9a-4e20-bea5-6d372ac8ab91" containerName="payload-pre-processing"
Oct 31 01:55:25.599326 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:25.599132    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="9feea519-15e9-4a7b-bd63-ab6bf4965707" containerName="payload-processing"
Oct 31 01:55:25.612516 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:25.605592    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba8822-rcwh7"
Oct 31 01:55:25.628267 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:25.622823    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddba8822-rcwh7"]
Oct 31 01:55:25.635596 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:25.635183    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e714d0b9-3aa2-4cce-9b1c-1e2f543164a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddba8822-rcwh7\" (UID: \"e714d0b9-3aa2-4cce-9b1c-1e2f543164a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba8822-rcwh7"
Oct 31 01:55:25.635596 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:25.635314    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e714d0b9-3aa2-4cce-9b1c-1e2f543164a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddba8822-rcwh7\" (UID: \"e714d0b9-3aa2-4cce-9b1c-1e2f543164a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba8822-rcwh7"
Oct 31 01:55:25.635596 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:25.635376    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m62vr\" (UniqueName: \"kubernetes.io/projected/e714d0b9-3aa2-4cce-9b1c-1e2f543164a6-kube-api-access-m62vr\") pod \"maas-api-revoke-sub-ddba8822-rcwh7\" (UID: \"e714d0b9-3aa2-4cce-9b1c-1e2f543164a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba8822-rcwh7"
Oct 31 01:55:25.636021 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:25.635833    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 01:55:25.636021 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:25.635906    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:55:25.736420 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:25.736376    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e714d0b9-3aa2-4cce-9b1c-1e2f543164a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddba8822-rcwh7\" (UID: \"e714d0b9-3aa2-4cce-9b1c-1e2f543164a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba8822-rcwh7"
Oct 31 01:55:25.736625 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:25.736598    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e714d0b9-3aa2-4cce-9b1c-1e2f543164a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddba8822-rcwh7\" (UID: \"e714d0b9-3aa2-4cce-9b1c-1e2f543164a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba8822-rcwh7"
Oct 31 01:55:25.736716 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:25.736688    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m62vr\" (UniqueName: \"kubernetes.io/projected/e714d0b9-3aa2-4cce-9b1c-1e2f543164a6-kube-api-access-m62vr\") pod \"maas-api-revoke-sub-ddba8822-rcwh7\" (UID: \"e714d0b9-3aa2-4cce-9b1c-1e2f543164a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba8822-rcwh7"
Oct 31 01:55:25.737343 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:25.737322    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e714d0b9-3aa2-4cce-9b1c-1e2f543164a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddba8822-rcwh7\" (UID: \"e714d0b9-3aa2-4cce-9b1c-1e2f543164a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba8822-rcwh7"
Oct 31 01:55:25.739513 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:25.739481    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e714d0b9-3aa2-4cce-9b1c-1e2f543164a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddba8822-rcwh7\" (UID: \"e714d0b9-3aa2-4cce-9b1c-1e2f543164a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba8822-rcwh7"
Oct 31 01:55:25.751217 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:25.751189    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m62vr\" (UniqueName: \"kubernetes.io/projected/e714d0b9-3aa2-4cce-9b1c-1e2f543164a6-kube-api-access-m62vr\") pod \"maas-api-revoke-sub-ddba8822-rcwh7\" (UID: \"e714d0b9-3aa2-4cce-9b1c-1e2f543164a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba8822-rcwh7"
Oct 31 01:55:25.945554 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:25.942118    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba8822-rcwh7"
Oct 31 01:55:26.310779 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:26.310747    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddba8822-rcwh7"]
Oct 31 01:55:26.311032 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:55:26.311005    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode714d0b9_3aa2_4cce_9b1c_1e2f543164a6.slice/crio-6940e1d422a9dcad67aff1a7b368b8bb117f7e41dd1482ee328f6eb87472ad53 WatchSource:0}: Error finding container 6940e1d422a9dcad67aff1a7b368b8bb117f7e41dd1482ee328f6eb87472ad53: Status 404 returned error can't find the container with id 6940e1d422a9dcad67aff1a7b368b8bb117f7e41dd1482ee328f6eb87472ad53
Oct 31 01:55:26.551463 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:26.537623    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba8822-rcwh7" event={"ID":"e714d0b9-3aa2-4cce-9b1c-1e2f543164a6","Type":"ContainerStarted","Data":"6940e1d422a9dcad67aff1a7b368b8bb117f7e41dd1482ee328f6eb87472ad53"}
Oct 31 01:55:27.549818 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:27.549731    5650 generic.go:356] "Generic (PLEG): container finished" podID="e714d0b9-3aa2-4cce-9b1c-1e2f543164a6" containerID="3d092f0b7025a6779715bb6434c11b263858fd053e958afc8bce16e6d170564c" exitCode=0
Oct 31 01:55:27.549818 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:27.549771    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba8822-rcwh7" event={"ID":"e714d0b9-3aa2-4cce-9b1c-1e2f543164a6","Type":"ContainerDied","Data":"3d092f0b7025a6779715bb6434c11b263858fd053e958afc8bce16e6d170564c"}
Oct 31 01:55:28.557423 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:55:28.556652    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:55:28.560149 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.560122    5650 generic.go:356] "Generic (PLEG): container finished" podID="7fe243de-3376-4d8d-87bd-032e29f0083d" containerID="87fce5c7e3a690205e8e1479f0bcf075a4a07c7e91f4a2f34c89863903a5fdc5" exitCode=0
Oct 31 01:55:28.560280 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.560193    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w" event={"ID":"7fe243de-3376-4d8d-87bd-032e29f0083d","Type":"ContainerDied","Data":"87fce5c7e3a690205e8e1479f0bcf075a4a07c7e91f4a2f34c89863903a5fdc5"}
Oct 31 01:55:28.560280 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.560259    5650 scope.go:122] "RemoveContainer" containerID="f7de8e8e314a30de871cf03eccd5bf5a7fd6b496e395ebc93be647095e74431a"
Oct 31 01:55:28.705227 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.705201    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:55:28.801586 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.801547    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-token\" (UniqueName: \"kubernetes.io/projected/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-token\") pod \"7fe243de-3376-4d8d-87bd-032e29f0083d\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") "
Oct 31 01:55:28.802023 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.801977    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fe243de-3376-4d8d-87bd-032e29f0083d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe243de-3376-4d8d-87bd-032e29f0083d-istiod-ca-cert\") pod \"7fe243de-3376-4d8d-87bd-032e29f0083d\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") "
Oct 31 01:55:28.802347 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.802308    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-podinfo\") pod \"7fe243de-3376-4d8d-87bd-032e29f0083d\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") "
Oct 31 01:55:28.802534 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.802516    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-envoy\") pod \"7fe243de-3376-4d8d-87bd-032e29f0083d\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") "
Oct 31 01:55:28.802707 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.802692    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-credential-socket\") pod \"7fe243de-3376-4d8d-87bd-032e29f0083d\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") "
Oct 31 01:55:28.802924 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.802908    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe243de-3376-4d8d-87bd-032e29f0083d-kube-api-access-xd7k8\" (UniqueName: \"kubernetes.io/projected/7fe243de-3376-4d8d-87bd-032e29f0083d-kube-api-access-xd7k8\") pod \"7fe243de-3376-4d8d-87bd-032e29f0083d\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") "
Oct 31 01:55:28.803057 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.803043    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-workload-socket\") pod \"7fe243de-3376-4d8d-87bd-032e29f0083d\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") "
Oct 31 01:55:28.803196 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.803181    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-data\") pod \"7fe243de-3376-4d8d-87bd-032e29f0083d\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") "
Oct 31 01:55:28.803477 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.803433    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-workload-certs\") pod \"7fe243de-3376-4d8d-87bd-032e29f0083d\" (UID: \"7fe243de-3376-4d8d-87bd-032e29f0083d\") "
Oct 31 01:55:28.815827 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.805161    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba8822-rcwh7"
Oct 31 01:55:28.815827 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.807679    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-workload-certs" pod "7fe243de-3376-4d8d-87bd-032e29f0083d" (UID: "7fe243de-3376-4d8d-87bd-032e29f0083d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:55:28.815827 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.808019    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-credential-socket" pod "7fe243de-3376-4d8d-87bd-032e29f0083d" (UID: "7fe243de-3376-4d8d-87bd-032e29f0083d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:55:28.815827 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.812516    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe243de-3376-4d8d-87bd-032e29f0083d-istiod-ca-cert" pod "7fe243de-3376-4d8d-87bd-032e29f0083d" (UID: "7fe243de-3376-4d8d-87bd-032e29f0083d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:55:28.815827 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.813002    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-workload-socket" pod "7fe243de-3376-4d8d-87bd-032e29f0083d" (UID: "7fe243de-3376-4d8d-87bd-032e29f0083d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:55:28.815827 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.813426    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-data" pod "7fe243de-3376-4d8d-87bd-032e29f0083d" (UID: "7fe243de-3376-4d8d-87bd-032e29f0083d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:55:28.815827 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.814991    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-podinfo" pod "7fe243de-3376-4d8d-87bd-032e29f0083d" (UID: "7fe243de-3376-4d8d-87bd-032e29f0083d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:55:28.818625 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.818594    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-envoy" pod "7fe243de-3376-4d8d-87bd-032e29f0083d" (UID: "7fe243de-3376-4d8d-87bd-032e29f0083d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:55:28.824638 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.824600    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe243de-3376-4d8d-87bd-032e29f0083d-kube-api-access-xd7k8" pod "7fe243de-3376-4d8d-87bd-032e29f0083d" (UID: "7fe243de-3376-4d8d-87bd-032e29f0083d"). InnerVolumeSpecName "kube-api-access-xd7k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:55:28.825810 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.825789    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-token" pod "7fe243de-3376-4d8d-87bd-032e29f0083d" (UID: "7fe243de-3376-4d8d-87bd-032e29f0083d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:55:28.907759 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.907720    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e714d0b9-3aa2-4cce-9b1c-1e2f543164a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e714d0b9-3aa2-4cce-9b1c-1e2f543164a6-maas-api-openshift-ca\") pod \"e714d0b9-3aa2-4cce-9b1c-1e2f543164a6\" (UID: \"e714d0b9-3aa2-4cce-9b1c-1e2f543164a6\") "
Oct 31 01:55:28.907910 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.907883    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e714d0b9-3aa2-4cce-9b1c-1e2f543164a6-kube-api-access-m62vr\" (UniqueName: \"kubernetes.io/projected/e714d0b9-3aa2-4cce-9b1c-1e2f543164a6-kube-api-access-m62vr\") pod \"e714d0b9-3aa2-4cce-9b1c-1e2f543164a6\" (UID: \"e714d0b9-3aa2-4cce-9b1c-1e2f543164a6\") "
Oct 31 01:55:28.908001 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.907976    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e714d0b9-3aa2-4cce-9b1c-1e2f543164a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e714d0b9-3aa2-4cce-9b1c-1e2f543164a6-maas-api-xks-ca\") pod \"e714d0b9-3aa2-4cce-9b1c-1e2f543164a6\" (UID: \"e714d0b9-3aa2-4cce-9b1c-1e2f543164a6\") "
Oct 31 01:55:28.908096 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.908071    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e714d0b9-3aa2-4cce-9b1c-1e2f543164a6-maas-api-openshift-ca" pod "e714d0b9-3aa2-4cce-9b1c-1e2f543164a6" (UID: "e714d0b9-3aa2-4cce-9b1c-1e2f543164a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:55:28.908491 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.908469    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:28.908491 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.908491    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:28.908636 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.908500    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:28.908636 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.908509    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e714d0b9-3aa2-4cce-9b1c-1e2f543164a6-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:28.908636 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.908518    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd7k8\" (UniqueName: \"kubernetes.io/projected/7fe243de-3376-4d8d-87bd-032e29f0083d-kube-api-access-xd7k8\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:28.908636 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.908526    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:28.908636 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.908534    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:28.908636 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.908543    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fe243de-3376-4d8d-87bd-032e29f0083d-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:28.908636 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.908551    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fe243de-3376-4d8d-87bd-032e29f0083d-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:28.908636 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.908561    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fe243de-3376-4d8d-87bd-032e29f0083d-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:28.909977 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.909954    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e714d0b9-3aa2-4cce-9b1c-1e2f543164a6-maas-api-xks-ca" pod "e714d0b9-3aa2-4cce-9b1c-1e2f543164a6" (UID: "e714d0b9-3aa2-4cce-9b1c-1e2f543164a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:55:28.910066 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:28.910038    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e714d0b9-3aa2-4cce-9b1c-1e2f543164a6-kube-api-access-m62vr" pod "e714d0b9-3aa2-4cce-9b1c-1e2f543164a6" (UID: "e714d0b9-3aa2-4cce-9b1c-1e2f543164a6"). InnerVolumeSpecName "kube-api-access-m62vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:55:29.010098 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:29.010057    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e714d0b9-3aa2-4cce-9b1c-1e2f543164a6-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:29.010098 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:29.010092    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m62vr\" (UniqueName: \"kubernetes.io/projected/e714d0b9-3aa2-4cce-9b1c-1e2f543164a6-kube-api-access-m62vr\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:29.570307 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:29.570274    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"
Oct 31 01:55:29.570738 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:29.570313    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w" event={"ID":"7fe243de-3376-4d8d-87bd-032e29f0083d","Type":"ContainerDied","Data":"1d12b9dff0656d1ccdda95e80804ba35e03d4a54b4c3f79837f3941dad47c3bc"}
Oct 31 01:55:29.570738 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:29.570386    5650 scope.go:122] "RemoveContainer" containerID="87fce5c7e3a690205e8e1479f0bcf075a4a07c7e91f4a2f34c89863903a5fdc5"
Oct 31 01:55:29.571944 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:29.571921    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba8822-rcwh7"
Oct 31 01:55:29.572065 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:29.571928    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddba8822-rcwh7" event={"ID":"e714d0b9-3aa2-4cce-9b1c-1e2f543164a6","Type":"ContainerDied","Data":"6940e1d422a9dcad67aff1a7b368b8bb117f7e41dd1482ee328f6eb87472ad53"}
Oct 31 01:55:29.572065 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:29.571985    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6940e1d422a9dcad67aff1a7b368b8bb117f7e41dd1482ee328f6eb87472ad53"
Oct 31 01:55:29.616072 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:29.616026    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"]
Oct 31 01:55:29.624431 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:29.624397    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-eb8ae6-openshift-default-7c44cd8cf7-vnj4w"]
Oct 31 01:55:30.119163 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.119125    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fe243de-3376-4d8d-87bd-032e29f0083d" path="/var/lib/kubelet/pods/7fe243de-3376-4d8d-87bd-032e29f0083d/volumes"
Oct 31 01:55:30.210671 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.210632    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"]
Oct 31 01:55:30.211674 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.211644    5650 cpu_manager.go:414] "containerMap: removing container" podUID="e714d0b9-3aa2-4cce-9b1c-1e2f543164a6" containerName="revoke-keys"
Oct 31 01:55:30.211674 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.211671    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e714d0b9-3aa2-4cce-9b1c-1e2f543164a6" containerName="revoke-keys"
Oct 31 01:55:30.211876 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.211693    5650 cpu_manager.go:414] "containerMap: removing container" podUID="7fe243de-3376-4d8d-87bd-032e29f0083d" containerName="istio-proxy"
Oct 31 01:55:30.211876 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.211701    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe243de-3376-4d8d-87bd-032e29f0083d" containerName="istio-proxy"
Oct 31 01:55:30.211876 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.211717    5650 cpu_manager.go:414] "containerMap: removing container" podUID="7fe243de-3376-4d8d-87bd-032e29f0083d" containerName="istio-proxy"
Oct 31 01:55:30.211876 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.211724    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe243de-3376-4d8d-87bd-032e29f0083d" containerName="istio-proxy"
Oct 31 01:55:30.212084 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.211890    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="e714d0b9-3aa2-4cce-9b1c-1e2f543164a6" containerName="revoke-keys"
Oct 31 01:55:30.212084 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.211906    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe243de-3376-4d8d-87bd-032e29f0083d" containerName="istio-proxy"
Oct 31 01:55:30.212084 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.211915    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe243de-3376-4d8d-87bd-032e29f0083d" containerName="istio-proxy"
Oct 31 01:55:30.218418 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.218388    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.232309 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.232281    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"]
Oct 31 01:55:30.238691 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.236052    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-2e6bfd-openshift-default-dockercfg-znd2f\""
Oct 31 01:55:30.331585 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.331525    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-token\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.331951 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.331929    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-credential-socket\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.333076 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.333052    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-podinfo\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.333329 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.333313    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-workload-socket\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.333555 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.333534    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-envoy\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.333801 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.333780    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-workload-certs\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.333973 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.333958    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-data\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.334180 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.334165    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istiod-ca-cert\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.334326 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.334308    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk5zv\" (UniqueName: \"kubernetes.io/projected/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-kube-api-access-gk5zv\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.436246 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.436150    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istiod-ca-cert\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.436246 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.436195    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk5zv\" (UniqueName: \"kubernetes.io/projected/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-kube-api-access-gk5zv\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.436508 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.436365    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-token\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.436508 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.436410    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-credential-socket\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.436694 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.436661    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-podinfo\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.436810 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.436765    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-workload-socket\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.436810 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.436771    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-credential-socket\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.436941 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.436797    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-envoy\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.436941 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.436888    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-workload-certs\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.436941 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.436933    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-data\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.437096 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.436963    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istiod-ca-cert\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.437288 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.437269    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-workload-socket\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.437373 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.437336    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-data\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.437415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.437400    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-workload-certs\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.439012 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.438992    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-envoy\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.439488 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.439467    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-podinfo\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.450092 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.450063    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-token\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.450192 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.450075    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk5zv\" (UniqueName: \"kubernetes.io/projected/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-kube-api-access-gk5zv\") pod \"e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") " pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.544722 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.544643    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:30.887575 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.887523    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"]
Oct 31 01:55:30.891523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.891481    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:55:30.891633 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.891578    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:55:30.891633 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.891622    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:55:30.989567 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.989533    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr"]
Oct 31 01:55:30.993807 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:30.993783    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr"
Oct 31 01:55:31.015470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:31.012702    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-7a850e\""
Oct 31 01:55:31.024009 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:31.023975    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-7a850e-dockercfg-bpg2q\""
Oct 31 01:55:31.041544 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:31.041501    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr"]
Oct 31 01:55:31.052496 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:31.052434    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw47z\" (UniqueName: \"kubernetes.io/projected/3d3621cc-41bb-4583-b189-ce7402a285cd-kube-api-access-jw47z\") pod \"payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr\" (UID: \"3d3621cc-41bb-4583-b189-ce7402a285cd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr"
Oct 31 01:55:31.052912 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:31.052866    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3621cc-41bb-4583-b189-ce7402a285cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr\" (UID: \"3d3621cc-41bb-4583-b189-ce7402a285cd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr"
Oct 31 01:55:31.079864 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:31.079832    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2"]
Oct 31 01:55:31.086581 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:31.086377    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2"
Oct 31 01:55:31.158631 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:31.158534    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3621cc-41bb-4583-b189-ce7402a285cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr\" (UID: \"3d3621cc-41bb-4583-b189-ce7402a285cd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr"
Oct 31 01:55:31.159012 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:31.158983    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw47z\" (UniqueName: \"kubernetes.io/projected/3d3621cc-41bb-4583-b189-ce7402a285cd-kube-api-access-jw47z\") pod \"payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr\" (UID: \"3d3621cc-41bb-4583-b189-ce7402a285cd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr"
Oct 31 01:55:31.191980 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:31.191940    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3621cc-41bb-4583-b189-ce7402a285cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr\" (UID: \"3d3621cc-41bb-4583-b189-ce7402a285cd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr"
Oct 31 01:55:31.256377 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:31.256297    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw47z\" (UniqueName: \"kubernetes.io/projected/3d3621cc-41bb-4583-b189-ce7402a285cd-kube-api-access-jw47z\") pod \"payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr\" (UID: \"3d3621cc-41bb-4583-b189-ce7402a285cd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr"
Oct 31 01:55:31.266473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:31.266349    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlq47\" (UniqueName: \"kubernetes.io/projected/8ac2ebbd-44b7-499a-be04-4d419002e32f-kube-api-access-dlq47\") pod \"payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2\" (UID: \"8ac2ebbd-44b7-499a-be04-4d419002e32f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2"
Oct 31 01:55:31.267216 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:31.266923    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac2ebbd-44b7-499a-be04-4d419002e32f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2\" (UID: \"8ac2ebbd-44b7-499a-be04-4d419002e32f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2"
Oct 31 01:55:31.287661 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:31.287553    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2"]
Oct 31 01:55:31.310114 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:31.309802    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr"
Oct 31 01:55:31.373945 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:31.373909    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac2ebbd-44b7-499a-be04-4d419002e32f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2\" (UID: \"8ac2ebbd-44b7-499a-be04-4d419002e32f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2"
Oct 31 01:55:31.374194 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:31.374172    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlq47\" (UniqueName: \"kubernetes.io/projected/8ac2ebbd-44b7-499a-be04-4d419002e32f-kube-api-access-dlq47\") pod \"payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2\" (UID: \"8ac2ebbd-44b7-499a-be04-4d419002e32f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2"
Oct 31 01:55:31.374909 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:31.374877    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac2ebbd-44b7-499a-be04-4d419002e32f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2\" (UID: \"8ac2ebbd-44b7-499a-be04-4d419002e32f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2"
Oct 31 01:55:31.390421 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:31.390384    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlq47\" (UniqueName: \"kubernetes.io/projected/8ac2ebbd-44b7-499a-be04-4d419002e32f-kube-api-access-dlq47\") pod \"payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2\" (UID: \"8ac2ebbd-44b7-499a-be04-4d419002e32f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2"
Oct 31 01:55:31.557612 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:31.557579    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2"
Oct 31 01:55:31.653832 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:31.649947    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8" event={"ID":"247b6d6e-faaf-44e3-bf73-73fbb620f5e0","Type":"ContainerStarted","Data":"c7c398c7b60890635bba29a9eb4f236353288f7452fc2dc48736905d4bb563b5"}
Oct 31 01:55:31.653832 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:31.650018    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8" event={"ID":"247b6d6e-faaf-44e3-bf73-73fbb620f5e0","Type":"ContainerStarted","Data":"bfa1a2d1d2ca7d7dfd8a9d2b07adbab73196d864a8001b3e42d08a9197032b77"}
Oct 31 01:55:31.675827 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:31.673281    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr"]
Oct 31 01:55:31.888940 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:31.888640    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8" podStartSLOduration=1.888621978 podStartE2EDuration="1.888621978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:31.693067847 +0000 UTC m=+2087.696782176" watchObservedRunningTime="2026-10-31 01:55:31.888621978 +0000 UTC m=+2087.892336269"
Oct 31 01:55:31.891771 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:31.891740    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2"]
Oct 31 01:55:31.892201 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:55:31.892179    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac2ebbd_44b7_499a_be04_4d419002e32f.slice/crio-cf470344a400f5e0edd3ca002700cd28eef350d0405b3209b8db36afb7cea36a WatchSource:0}: Error finding container cf470344a400f5e0edd3ca002700cd28eef350d0405b3209b8db36afb7cea36a: Status 404 returned error can't find the container with id cf470344a400f5e0edd3ca002700cd28eef350d0405b3209b8db36afb7cea36a
Oct 31 01:55:32.544756 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:32.544723    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:32.556935 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:32.552964    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:32.659773 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:32.659736    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr" event={"ID":"3d3621cc-41bb-4583-b189-ce7402a285cd","Type":"ContainerStarted","Data":"ef5b8aad127c1dff1cc7462a1904495fc686f2e64d18ba9297fd686da26758ff"}
Oct 31 01:55:32.659966 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:32.659792    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr" event={"ID":"3d3621cc-41bb-4583-b189-ce7402a285cd","Type":"ContainerStarted","Data":"b736644c6e7ac49868ffd1ce2da5231acd11056b58995933921ba7a8a4eba87c"}
Oct 31 01:55:32.659966 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:32.659816    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr"
Oct 31 01:55:32.661684 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:32.661657    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2" event={"ID":"8ac2ebbd-44b7-499a-be04-4d419002e32f","Type":"ContainerStarted","Data":"1274a09d74b8206892ef582528e6f6de9b37ce25c728938292c30775e2aa1ef3"}
Oct 31 01:55:32.661837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:32.661715    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2"
Oct 31 01:55:32.661837 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:32.661740    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2" event={"ID":"8ac2ebbd-44b7-499a-be04-4d419002e32f","Type":"ContainerStarted","Data":"cf470344a400f5e0edd3ca002700cd28eef350d0405b3209b8db36afb7cea36a"}
Oct 31 01:55:32.661917 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:32.661853    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:32.662884 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:32.662865    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:55:32.686128 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:32.686068    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr" podStartSLOduration=2.6860530430000003 podStartE2EDuration="2.686053043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:32.682762727 +0000 UTC m=+2088.686477018" watchObservedRunningTime="2026-10-31 01:55:32.686053043 +0000 UTC m=+2088.689767333"
Oct 31 01:55:34.147842 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:34.147784    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2" podStartSLOduration=3.147769434 podStartE2EDuration="3.147769434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:32.731769207 +0000 UTC m=+2088.735483501" watchObservedRunningTime="2026-10-31 01:55:34.147769434 +0000 UTC m=+2090.151483763"
Oct 31 01:55:34.150200 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:34.150176    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh"]
Oct 31 01:55:35.476651 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:35.476614    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw"]
Oct 31 01:55:35.481405 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:35.481379    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw"
Oct 31 01:55:35.489290 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:35.489259    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw"]
Oct 31 01:55:35.497498 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:35.497435    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-2e6bfd\""
Oct 31 01:55:35.562775 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:35.562736    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j5pl\" (UniqueName: \"kubernetes.io/projected/459bd3ef-75c0-4b05-baa1-90deddc2d4ab-kube-api-access-4j5pl\") pod \"payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw\" (UID: \"459bd3ef-75c0-4b05-baa1-90deddc2d4ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw"
Oct 31 01:55:35.562999 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:35.562976    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459bd3ef-75c0-4b05-baa1-90deddc2d4ab-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw\" (UID: \"459bd3ef-75c0-4b05-baa1-90deddc2d4ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw"
Oct 31 01:55:35.670468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:35.667570    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459bd3ef-75c0-4b05-baa1-90deddc2d4ab-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw\" (UID: \"459bd3ef-75c0-4b05-baa1-90deddc2d4ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw"
Oct 31 01:55:35.670468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:35.667758    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j5pl\" (UniqueName: \"kubernetes.io/projected/459bd3ef-75c0-4b05-baa1-90deddc2d4ab-kube-api-access-4j5pl\") pod \"payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw\" (UID: \"459bd3ef-75c0-4b05-baa1-90deddc2d4ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw"
Oct 31 01:55:35.670468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:35.669015    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459bd3ef-75c0-4b05-baa1-90deddc2d4ab-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw\" (UID: \"459bd3ef-75c0-4b05-baa1-90deddc2d4ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw"
Oct 31 01:55:35.744408 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:35.744327    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j5pl\" (UniqueName: \"kubernetes.io/projected/459bd3ef-75c0-4b05-baa1-90deddc2d4ab-kube-api-access-4j5pl\") pod \"payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw\" (UID: \"459bd3ef-75c0-4b05-baa1-90deddc2d4ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw"
Oct 31 01:55:35.802389 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:35.802355    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw"
Oct 31 01:55:36.095772 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.095744    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw"]
Oct 31 01:55:36.096376 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:55:36.096348    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459bd3ef_75c0_4b05_baa1_90deddc2d4ab.slice/crio-1784610b234d3ff2984b5eef9405551da23051d523e4593a99f1e7839bdbb732 WatchSource:0}: Error finding container 1784610b234d3ff2984b5eef9405551da23051d523e4593a99f1e7839bdbb732: Status 404 returned error can't find the container with id 1784610b234d3ff2984b5eef9405551da23051d523e4593a99f1e7839bdbb732
Oct 31 01:55:36.490248 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.490208    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"]
Oct 31 01:55:36.495180 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.495150    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.520135 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.520098    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"]
Oct 31 01:55:36.609700 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.609655    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.609866 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.609785    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.609911 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.609865    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.609911 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.609898    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.610003 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.609986    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.610110 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.610098    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61e54f0-3295-4e02-aceb-2190f5b00181-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.610153 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.610142    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.610190 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.610167    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.610245 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.610227    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znjcp\" (UniqueName: \"kubernetes.io/projected/b61e54f0-3295-4e02-aceb-2190f5b00181-kube-api-access-znjcp\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.711828 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.711778    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61e54f0-3295-4e02-aceb-2190f5b00181-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.711985 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.711879    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.711985 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.711920    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.711985 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.711968    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znjcp\" (UniqueName: \"kubernetes.io/projected/b61e54f0-3295-4e02-aceb-2190f5b00181-kube-api-access-znjcp\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.712171 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.712034    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.712171 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.712142    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.712270 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.712190    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.712270 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.712237    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.712375 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.712306    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.712375 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.712338    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.712638 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.712608    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61e54f0-3295-4e02-aceb-2190f5b00181-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.712774 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.712656    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.712774 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.712728    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.712908 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.712788    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.714675 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.714651    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.714882 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.714865    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.728151 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.728114    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.728346 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.728329    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znjcp\" (UniqueName: \"kubernetes.io/projected/b61e54f0-3295-4e02-aceb-2190f5b00181-kube-api-access-znjcp\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:36.746440 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.746317    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw" event={"ID":"459bd3ef-75c0-4b05-baa1-90deddc2d4ab","Type":"ContainerStarted","Data":"3efbf4b96d3c399c60c66962bb1969fa6d8aee9170c3766ae98c95e197b27e54"}
Oct 31 01:55:36.746440 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.746381    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw" event={"ID":"459bd3ef-75c0-4b05-baa1-90deddc2d4ab","Type":"ContainerStarted","Data":"1784610b234d3ff2984b5eef9405551da23051d523e4593a99f1e7839bdbb732"}
Oct 31 01:55:36.746682 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.746606    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw"
Oct 31 01:55:36.767502 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.767425    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw" podStartSLOduration=1.767407842 podStartE2EDuration="1.767407842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:36.765980179 +0000 UTC m=+2092.769694493" watchObservedRunningTime="2026-10-31 01:55:36.767407842 +0000 UTC m=+2092.771122135"
Oct 31 01:55:36.832907 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:36.828072    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:37.388415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:37.380508    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:55:37.388415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:37.380602    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:55:37.388415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:37.380647    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:55:37.388415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:37.383060    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"]
Oct 31 01:55:37.768882 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:37.768844    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx" event={"ID":"b61e54f0-3295-4e02-aceb-2190f5b00181","Type":"ContainerStarted","Data":"51f48d9755b14f29591aa6392f7dba98fa71d562466cdf99fab2710729c084a0"}
Oct 31 01:55:37.769493 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:37.769464    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx" event={"ID":"b61e54f0-3295-4e02-aceb-2190f5b00181","Type":"ContainerStarted","Data":"ae0d92e6b2bfa150f5e4e35a6f85a838f2e41c51d16001e3edfd432430df5832"}
Oct 31 01:55:37.799209 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:37.799152    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx" podStartSLOduration=1.7991356459999999 podStartE2EDuration="1.799135646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:37.796835718 +0000 UTC m=+2093.800550041" watchObservedRunningTime="2026-10-31 01:55:37.799135646 +0000 UTC m=+2093.802849936"
Oct 31 01:55:37.833489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:37.833061    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:38.865474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:38.864743    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 31 01:55:38.865474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:38.864805    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx" podUID="b61e54f0-3295-4e02-aceb-2190f5b00181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 31 01:55:39.156528 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:39.152770    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh" podUID="26b7a731-46d2-4702-94c3-56ed4ad79399" containerName="payload-processing" containerID="cri-o://ea66690969c56278e1c3ac545d1b8c2eddcd2ee03ebba1dfda555cfd8dea72a1" gracePeriod=25
Oct 31 01:55:40.125662 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:40.125621    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 31 01:55:40.126118 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:40.125689    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx" podUID="b61e54f0-3295-4e02-aceb-2190f5b00181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 31 01:55:40.418558 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:40.413116    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh"
Oct 31 01:55:40.473474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:40.458135    5650 generic.go:356] "Generic (PLEG): container finished" podID="26b7a731-46d2-4702-94c3-56ed4ad79399" containerID="ea66690969c56278e1c3ac545d1b8c2eddcd2ee03ebba1dfda555cfd8dea72a1" exitCode=0
Oct 31 01:55:40.498429 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:40.487701    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh" event={"ID":"26b7a731-46d2-4702-94c3-56ed4ad79399","Type":"ContainerDied","Data":"ea66690969c56278e1c3ac545d1b8c2eddcd2ee03ebba1dfda555cfd8dea72a1"}
Oct 31 01:55:40.498429 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:40.487779    5650 scope.go:122] "RemoveContainer" containerID="ea66690969c56278e1c3ac545d1b8c2eddcd2ee03ebba1dfda555cfd8dea72a1"
Oct 31 01:55:40.519472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:40.516047    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26b7a731-46d2-4702-94c3-56ed4ad79399-kube-api-access-th96c\" (UniqueName: \"kubernetes.io/projected/26b7a731-46d2-4702-94c3-56ed4ad79399-kube-api-access-th96c\") pod \"26b7a731-46d2-4702-94c3-56ed4ad79399\" (UID: \"26b7a731-46d2-4702-94c3-56ed4ad79399\") "
Oct 31 01:55:40.519472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:40.516361    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26b7a731-46d2-4702-94c3-56ed4ad79399-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b7a731-46d2-4702-94c3-56ed4ad79399-plugins-config-volume\") pod \"26b7a731-46d2-4702-94c3-56ed4ad79399\" (UID: \"26b7a731-46d2-4702-94c3-56ed4ad79399\") "
Oct 31 01:55:40.573470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:40.563342    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b7a731-46d2-4702-94c3-56ed4ad79399-plugins-config-volume" pod "26b7a731-46d2-4702-94c3-56ed4ad79399" (UID: "26b7a731-46d2-4702-94c3-56ed4ad79399"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:55:40.591611 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:40.589638    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b7a731-46d2-4702-94c3-56ed4ad79399-kube-api-access-th96c" pod "26b7a731-46d2-4702-94c3-56ed4ad79399" (UID: "26b7a731-46d2-4702-94c3-56ed4ad79399"). InnerVolumeSpecName "kube-api-access-th96c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:55:40.684302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:40.675106    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26b7a731-46d2-4702-94c3-56ed4ad79399-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:40.684302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:40.675149    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th96c\" (UniqueName: \"kubernetes.io/projected/26b7a731-46d2-4702-94c3-56ed4ad79399-kube-api-access-th96c\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:55:40.835142 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:40.835103    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 31 01:55:40.835412 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:40.835376    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx" podUID="b61e54f0-3295-4e02-aceb-2190f5b00181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 31 01:55:41.146974 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:41.146916    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh" podUID="26b7a731-46d2-4702-94c3-56ed4ad79399" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.155:50052\" within 1s: context deadline exceeded"
Oct 31 01:55:41.146974 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:55:41.146978    5650 logging.go:55] [core] [Channel #1791 SubChannel #1792]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.155:50052", ServerName: "10.134.0.155:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.155:50052: operation was canceled"
Oct 31 01:55:41.470357 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:41.470316    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh" event={"ID":"26b7a731-46d2-4702-94c3-56ed4ad79399","Type":"ContainerDied","Data":"6cd4f26d1075d6413b0fcff2bbbff7e75cd829b35e0cd10ac0e33520fa3f1d34"}
Oct 31 01:55:41.470550 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:41.470494    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh"
Oct 31 01:55:41.531095 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:41.529122    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh"]
Oct 31 01:55:41.533887 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:41.533844    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-dbff33-5d5dfd769c-l7jmh"]
Oct 31 01:55:41.834253 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:41.834141    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 31 01:55:41.834253 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:41.834217    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx" podUID="b61e54f0-3295-4e02-aceb-2190f5b00181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 31 01:55:42.130677 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:42.130594    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26b7a731-46d2-4702-94c3-56ed4ad79399" path="/var/lib/kubelet/pods/26b7a731-46d2-4702-94c3-56ed4ad79399/volumes"
Oct 31 01:55:42.833886 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:42.833844    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 31 01:55:42.834359 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:42.833922    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx" podUID="b61e54f0-3295-4e02-aceb-2190f5b00181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 31 01:55:43.672066 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:43.672039    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr"
Oct 31 01:55:43.672459 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:43.672428    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2"
Oct 31 01:55:43.833849 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:43.833816    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 31 01:55:43.834017 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:43.833884    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx" podUID="b61e54f0-3295-4e02-aceb-2190f5b00181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 31 01:55:43.841825 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:43.841794    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w"]
Oct 31 01:55:43.842456 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:43.842431    5650 cpu_manager.go:414] "containerMap: removing container" podUID="26b7a731-46d2-4702-94c3-56ed4ad79399" containerName="payload-processing"
Oct 31 01:55:43.842501 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:43.842458    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b7a731-46d2-4702-94c3-56ed4ad79399" containerName="payload-processing"
Oct 31 01:55:43.842629 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:43.842619    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b7a731-46d2-4702-94c3-56ed4ad79399" containerName="payload-processing"
Oct 31 01:55:43.847225 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:43.847203    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w"
Oct 31 01:55:43.856212 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:43.856188    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w"]
Oct 31 01:55:43.861832 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:43.861802    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-6d7f5d\""
Oct 31 01:55:43.861993 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:43.861876    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-6d7f5d-dockercfg-7hfz5\""
Oct 31 01:55:43.956578 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:43.953749    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55d0de06-2458-43fb-bf56-f02492a67dea-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w\" (UID: \"55d0de06-2458-43fb-bf56-f02492a67dea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w"
Oct 31 01:55:43.956578 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:43.953802    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8psp\" (UniqueName: \"kubernetes.io/projected/55d0de06-2458-43fb-bf56-f02492a67dea-kube-api-access-q8psp\") pod \"payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w\" (UID: \"55d0de06-2458-43fb-bf56-f02492a67dea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w"
Oct 31 01:55:44.068469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:44.062936    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55d0de06-2458-43fb-bf56-f02492a67dea-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w\" (UID: \"55d0de06-2458-43fb-bf56-f02492a67dea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w"
Oct 31 01:55:44.068469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:44.062994    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8psp\" (UniqueName: \"kubernetes.io/projected/55d0de06-2458-43fb-bf56-f02492a67dea-kube-api-access-q8psp\") pod \"payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w\" (UID: \"55d0de06-2458-43fb-bf56-f02492a67dea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w"
Oct 31 01:55:44.068469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:44.064221    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55d0de06-2458-43fb-bf56-f02492a67dea-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w\" (UID: \"55d0de06-2458-43fb-bf56-f02492a67dea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w"
Oct 31 01:55:44.143692 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:44.143658    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8psp\" (UniqueName: \"kubernetes.io/projected/55d0de06-2458-43fb-bf56-f02492a67dea-kube-api-access-q8psp\") pod \"payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w\" (UID: \"55d0de06-2458-43fb-bf56-f02492a67dea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w"
Oct 31 01:55:44.175809 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:44.175765    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-6d7f5d-dockercfg-7hfz5\""
Oct 31 01:55:44.181177 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:44.181141    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w"
Oct 31 01:55:44.526537 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:44.526502    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w"]
Oct 31 01:55:44.834483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:44.834365    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 31 01:55:44.834483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:44.834421    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx" podUID="b61e54f0-3295-4e02-aceb-2190f5b00181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 31 01:55:44.982229 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:44.981990    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"]
Oct 31 01:55:44.987525 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:44.987501    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.005527 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.005492    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"]
Oct 31 01:55:45.059325 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.059290    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-6c54a1-openshift-default-dockercfg-mzwh5\""
Oct 31 01:55:45.126526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.123553    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-podinfo\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.126526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.123641    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-workload-socket\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.126526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.123716    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d39c057-51e1-49f1-85de-facd9e8f15fc-istiod-ca-cert\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.126526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.123759    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-token\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.126526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.123871    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-credential-socket\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.126526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.123924    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-data\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.126526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.124000    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm282\" (UniqueName: \"kubernetes.io/projected/0d39c057-51e1-49f1-85de-facd9e8f15fc-kube-api-access-nm282\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.126526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.124061    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-envoy\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.126526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.124143    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-workload-certs\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.232481 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.229439    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-podinfo\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.232481 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.229538    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-workload-socket\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.232481 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.229611    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d39c057-51e1-49f1-85de-facd9e8f15fc-istiod-ca-cert\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.232481 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.229657    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-token\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.232481 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.229767    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-credential-socket\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.232481 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.229818    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-data\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.232481 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.229888    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm282\" (UniqueName: \"kubernetes.io/projected/0d39c057-51e1-49f1-85de-facd9e8f15fc-kube-api-access-nm282\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.232481 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.229948    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-envoy\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.232481 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.230027    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-workload-certs\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.232481 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.230599    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-workload-certs\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.232481 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.231212    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-credential-socket\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.232481 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.231475    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-workload-socket\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.232481 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.232142    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d39c057-51e1-49f1-85de-facd9e8f15fc-istiod-ca-cert\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.233268 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.232966    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-data\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.295310 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.295242    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-token\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.297317 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.297203    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-podinfo\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.308322 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.308281    5650 scope.go:122] "RemoveContainer" containerID="b0425641d7c3e62116c31d4cf3feca2ef23c72081857bdb6c75170da706a43d0"
Oct 31 01:55:45.311459 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.310523    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-envoy\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.315298 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.315264    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm282\" (UniqueName: \"kubernetes.io/projected/0d39c057-51e1-49f1-85de-facd9e8f15fc-kube-api-access-nm282\") pod \"e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") " pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.337195 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.337161    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:45.533760 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.533710    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w" event={"ID":"55d0de06-2458-43fb-bf56-f02492a67dea","Type":"ContainerStarted","Data":"19bc8fb5dd527d4ffe3b73062983a8a86032c5e5a371aa6b77f2d05e582d6fe5"}
Oct 31 01:55:45.533967 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.533784    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w" event={"ID":"55d0de06-2458-43fb-bf56-f02492a67dea","Type":"ContainerStarted","Data":"b9e7ce6bb425952c040bf9c67936649c67356ea748b99b24a95fca7eb5504810"}
Oct 31 01:55:45.533967 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.533820    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w"
Oct 31 01:55:45.558668 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.557878    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w" podStartSLOduration=2.557855067 podStartE2EDuration="2.557855067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:45.553577931 +0000 UTC m=+2101.557292224" watchObservedRunningTime="2026-10-31 01:55:45.557855067 +0000 UTC m=+2101.561569360"
Oct 31 01:55:45.682913 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.682846    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"]
Oct 31 01:55:45.688346 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:55:45.688290    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d39c057_51e1_49f1_85de_facd9e8f15fc.slice/crio-3ee1c0ed1d608bd838989a4835b814956efdb1e729c494a2361d90f1aefa8693 WatchSource:0}: Error finding container 3ee1c0ed1d608bd838989a4835b814956efdb1e729c494a2361d90f1aefa8693: Status 404 returned error can't find the container with id 3ee1c0ed1d608bd838989a4835b814956efdb1e729c494a2361d90f1aefa8693
Oct 31 01:55:45.690962 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.690915    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:55:45.691091 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.691010    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:55:45.691091 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:45.691056    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:55:46.546050 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:46.545837    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np" event={"ID":"0d39c057-51e1-49f1-85de-facd9e8f15fc","Type":"ContainerStarted","Data":"59a60e4b44b4c5a3fb2fafb89a0f33421e9c07de1142c2dd75c40fb56d727ddd"}
Oct 31 01:55:46.546050 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:46.545999    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np" event={"ID":"0d39c057-51e1-49f1-85de-facd9e8f15fc","Type":"ContainerStarted","Data":"3ee1c0ed1d608bd838989a4835b814956efdb1e729c494a2361d90f1aefa8693"}
Oct 31 01:55:46.579273 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:46.579216    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np" podStartSLOduration=2.57920141 podStartE2EDuration="2.57920141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:46.575992737 +0000 UTC m=+2102.579707041" watchObservedRunningTime="2026-10-31 01:55:46.57920141 +0000 UTC m=+2102.582915762"
Oct 31 01:55:46.879626 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:46.879534    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:55:46.879913 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:46.879868    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx" podUID="b61e54f0-3295-4e02-aceb-2190f5b00181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:47.291570 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:47.291529    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:47.292950 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:47.292918    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:47.294594 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:47.294561    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 01:55:47.338364 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:47.338327    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:47.344818 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:47.344786    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:47.554034 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:47.553941    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:47.554875 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:47.554852    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:55:47.771225 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:47.771195    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw"
Oct 31 01:55:50.665258 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.665219    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"]
Oct 31 01:55:50.671069 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.671044    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.671529 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.671500    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"]
Oct 31 01:55:50.676049 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.676023    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-ae5a1c-openshift-default-dockercfg-ctkcr\""
Oct 31 01:55:50.765469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.761603    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-podinfo\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.765469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.761672    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-workload-certs\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.765469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.761768    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-workload-socket\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.765469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.761806    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-credential-socket\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.765469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.761896    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-envoy\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.765469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.761991    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-data\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.765469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.762072    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-token\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.765469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.762117    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcdv9\" (UniqueName: \"kubernetes.io/projected/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-kube-api-access-rcdv9\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.765469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.762143    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.864467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.864021    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-data\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.864467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.864149    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-token\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.864467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.864277    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcdv9\" (UniqueName: \"kubernetes.io/projected/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-kube-api-access-rcdv9\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.864467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.864325    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.865154 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.864544    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-podinfo\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.865277 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.865219    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-workload-certs\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.865352 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.865331    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-workload-socket\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.865407 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.865384    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-credential-socket\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.865551 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.865529    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-envoy\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.865811 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.865784    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-data\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.865944 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.865807    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-workload-certs\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.866076 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.866056    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-workload-socket\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.866366 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.866339    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-credential-socket\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.866547 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.866522    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.867886 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.867867    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-podinfo\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.868861 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.868839    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-envoy\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.883467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.882614    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcdv9\" (UniqueName: \"kubernetes.io/projected/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-kube-api-access-rcdv9\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.883467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.882617    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-token\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:50.990635 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:50.990599    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:51.291601 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:51.291568    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"]
Oct 31 01:55:51.292163 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:55:51.292134    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93cf4d0_6dd7_40b5_ab25_a6ba6dc964db.slice/crio-67066061c7e9ac2e4022e2ed8ea9aaf658ed691d69d785d3810325dec68c9a69 WatchSource:0}: Error finding container 67066061c7e9ac2e4022e2ed8ea9aaf658ed691d69d785d3810325dec68c9a69: Status 404 returned error can't find the container with id 67066061c7e9ac2e4022e2ed8ea9aaf658ed691d69d785d3810325dec68c9a69
Oct 31 01:55:51.295393 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:51.295353    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:55:51.295537 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:51.295460    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:55:51.295537 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:51.295505    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:55:51.612306 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:51.612208    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf" event={"ID":"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db","Type":"ContainerStarted","Data":"a40a5f0f3953f38da44794fc7ab5fce6c1efea4f791b8205452a6b6d054beb6d"}
Oct 31 01:55:51.612306 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:51.612271    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf" event={"ID":"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db","Type":"ContainerStarted","Data":"67066061c7e9ac2e4022e2ed8ea9aaf658ed691d69d785d3810325dec68c9a69"}
Oct 31 01:55:51.638979 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:51.638905    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf" podStartSLOduration=1.638885744 podStartE2EDuration="1.638885744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:51.637178501 +0000 UTC m=+2107.640892794" watchObservedRunningTime="2026-10-31 01:55:51.638885744 +0000 UTC m=+2107.642600036"
Oct 31 01:55:51.991363 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:51.991329    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:52.997646 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:52.997608    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:53.672866 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:53.672830    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:53.673689 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:53.673669    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:55:54.920389 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:54.920352    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"]
Oct 31 01:55:54.920964 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:54.920648    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z" podUID="f6011af8-136f-436a-bb13-987d5844f336" containerName="istio-proxy" containerID="cri-o://5308604b5f247586a17fad9cb72a29828c5cab523852a19adae7b777b07f7f6a" gracePeriod=30
Oct 31 01:55:55.398623 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:55.398585    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l"]
Oct 31 01:55:55.403461 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:55.403428    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l"
Oct 31 01:55:55.408661 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:55.408617    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-ae5a1c\""
Oct 31 01:55:55.409717 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:55.409675    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-ae5a1c-dockercfg-sshzj\""
Oct 31 01:55:55.415935 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:55.415903    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l"]
Oct 31 01:55:55.458858 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:55.458805    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tln6f\" (UniqueName: \"kubernetes.io/projected/8969a12d-73b4-4d52-a144-918c72ded4bc-kube-api-access-tln6f\") pod \"payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l\" (UID: \"8969a12d-73b4-4d52-a144-918c72ded4bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l"
Oct 31 01:55:55.459016 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:55.458990    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8969a12d-73b4-4d52-a144-918c72ded4bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l\" (UID: \"8969a12d-73b4-4d52-a144-918c72ded4bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l"
Oct 31 01:55:55.603469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:55.598119    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tln6f\" (UniqueName: \"kubernetes.io/projected/8969a12d-73b4-4d52-a144-918c72ded4bc-kube-api-access-tln6f\") pod \"payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l\" (UID: \"8969a12d-73b4-4d52-a144-918c72ded4bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l"
Oct 31 01:55:55.603469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:55.598263    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8969a12d-73b4-4d52-a144-918c72ded4bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l\" (UID: \"8969a12d-73b4-4d52-a144-918c72ded4bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l"
Oct 31 01:55:55.611475 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:55.609098    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8969a12d-73b4-4d52-a144-918c72ded4bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l\" (UID: \"8969a12d-73b4-4d52-a144-918c72ded4bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l"
Oct 31 01:55:55.683764 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:55.683492    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tln6f\" (UniqueName: \"kubernetes.io/projected/8969a12d-73b4-4d52-a144-918c72ded4bc-kube-api-access-tln6f\") pod \"payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l\" (UID: \"8969a12d-73b4-4d52-a144-918c72ded4bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l"
Oct 31 01:55:55.739468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:55.732897    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l"
Oct 31 01:55:56.035815 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:55:56.035772    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8969a12d_73b4_4d52_a144_918c72ded4bc.slice/crio-8c1b9e02b20a21c9f6561c17bf811260e1501dff1b78a69d635ab7ed9f2041ff WatchSource:0}: Error finding container 8c1b9e02b20a21c9f6561c17bf811260e1501dff1b78a69d635ab7ed9f2041ff: Status 404 returned error can't find the container with id 8c1b9e02b20a21c9f6561c17bf811260e1501dff1b78a69d635ab7ed9f2041ff
Oct 31 01:55:56.036775 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:56.036746    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l"]
Oct 31 01:55:56.553886 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:56.553854    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w"
Oct 31 01:55:56.727439 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:56.727396    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l" event={"ID":"8969a12d-73b4-4d52-a144-918c72ded4bc","Type":"ContainerStarted","Data":"15ac7aeaaeaadde58c127eb460dccae3f1b72473c0f6086f85769ba1a5e9d298"}
Oct 31 01:55:56.727655 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:56.727486    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l"
Oct 31 01:55:56.727655 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:56.727513    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l" event={"ID":"8969a12d-73b4-4d52-a144-918c72ded4bc","Type":"ContainerStarted","Data":"8c1b9e02b20a21c9f6561c17bf811260e1501dff1b78a69d635ab7ed9f2041ff"}
Oct 31 01:55:56.752040 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:56.751969    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l" podStartSLOduration=1.751953618 podStartE2EDuration="1.751953618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:56.748584629 +0000 UTC m=+2112.752298921" watchObservedRunningTime="2026-10-31 01:55:56.751953618 +0000 UTC m=+2112.755667908"
Oct 31 01:55:58.443551 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.443510    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"]
Oct 31 01:55:58.462598 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.461053    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"]
Oct 31 01:55:58.462598 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.461192    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.676469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.670204    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.676469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.670306    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.676469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.670342    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.676469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.670458    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.676469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.670537    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.676469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.670575    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.676469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.670653    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.676469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.670701    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.676469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.670763    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b94b\" (UniqueName: \"kubernetes.io/projected/6aa3fc04-120c-4dd7-ba46-99b133d092b0-kube-api-access-6b94b\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.767578 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:55:58.766944    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:55:58.772565 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.772530    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.772735 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.772589    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.772735 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.772665    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.772735 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.772716    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.772921 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.772778    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b94b\" (UniqueName: \"kubernetes.io/projected/6aa3fc04-120c-4dd7-ba46-99b133d092b0-kube-api-access-6b94b\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.773050 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.773021    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.773145 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.773106    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.773145 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.773127    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.773242 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.773148    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.773296 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.773276    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.773652 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.773627    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.773994 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.773966    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.774355 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.774330    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.775982 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.775957    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.778276 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.778255    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.778505 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.778485    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.797600 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.797573    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.797750 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.797662    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b94b\" (UniqueName: \"kubernetes.io/projected/6aa3fc04-120c-4dd7-ba46-99b133d092b0-kube-api-access-6b94b\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:58.849645 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.849609    5650 patch_prober.go:29] interesting pod/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:55:58.850272 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.849945    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z" podUID="f6011af8-136f-436a-bb13-987d5844f336" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:55:58.945472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:58.933743    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:55:59.397852 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:59.397751    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:55:59.397852 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:59.397848    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:55:59.398106 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:59.397889    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:55:59.425708 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:59.420571    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"]
Oct 31 01:55:59.856206 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:59.853050    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2" event={"ID":"6aa3fc04-120c-4dd7-ba46-99b133d092b0","Type":"ContainerStarted","Data":"7c77695d20de8042934cc6056a815aa4fdb7fdc9dcf2b6a2a65c0eebfc676bfc"}
Oct 31 01:55:59.856206 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:59.853131    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2" event={"ID":"6aa3fc04-120c-4dd7-ba46-99b133d092b0","Type":"ContainerStarted","Data":"28e2259b542bf6ad370aa37efaed1103a3a8e5967ff16d3caffa1a5f005aef2b"}
Oct 31 01:55:59.884042 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:59.883944    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2" podStartSLOduration=1.8839246429999998 podStartE2EDuration="1.883924643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:59.879977592 +0000 UTC m=+2115.883691883" watchObservedRunningTime="2026-10-31 01:55:59.883924643 +0000 UTC m=+2115.887638935"
Oct 31 01:55:59.943744 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:55:59.940129    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:56:00.687252 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.682777    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:56:00.784041 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.782989    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-istio-data\") pod \"f6011af8-136f-436a-bb13-987d5844f336\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") "
Oct 31 01:56:00.784041 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.783068    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6011af8-136f-436a-bb13-987d5844f336-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6011af8-136f-436a-bb13-987d5844f336-istio-podinfo\") pod \"f6011af8-136f-436a-bb13-987d5844f336\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") "
Oct 31 01:56:00.784041 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.783148    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-istio-envoy\") pod \"f6011af8-136f-436a-bb13-987d5844f336\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") "
Oct 31 01:56:00.784041 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.783184    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-workload-socket\") pod \"f6011af8-136f-436a-bb13-987d5844f336\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") "
Oct 31 01:56:00.784041 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.783251    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6011af8-136f-436a-bb13-987d5844f336-istio-token\" (UniqueName: \"kubernetes.io/projected/f6011af8-136f-436a-bb13-987d5844f336-istio-token\") pod \"f6011af8-136f-436a-bb13-987d5844f336\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") "
Oct 31 01:56:00.784041 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.783311    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-credential-socket\") pod \"f6011af8-136f-436a-bb13-987d5844f336\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") "
Oct 31 01:56:00.784041 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.783345    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6011af8-136f-436a-bb13-987d5844f336-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6011af8-136f-436a-bb13-987d5844f336-istiod-ca-cert\") pod \"f6011af8-136f-436a-bb13-987d5844f336\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") "
Oct 31 01:56:00.784041 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.783538    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-workload-certs\") pod \"f6011af8-136f-436a-bb13-987d5844f336\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") "
Oct 31 01:56:00.784041 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.783585    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6011af8-136f-436a-bb13-987d5844f336-kube-api-access-ck2jv\" (UniqueName: \"kubernetes.io/projected/f6011af8-136f-436a-bb13-987d5844f336-kube-api-access-ck2jv\") pod \"f6011af8-136f-436a-bb13-987d5844f336\" (UID: \"f6011af8-136f-436a-bb13-987d5844f336\") "
Oct 31 01:56:00.785111 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.784931    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-istio-data" pod "f6011af8-136f-436a-bb13-987d5844f336" (UID: "f6011af8-136f-436a-bb13-987d5844f336"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:56:00.786344 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.785263    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6011af8-136f-436a-bb13-987d5844f336-istiod-ca-cert" pod "f6011af8-136f-436a-bb13-987d5844f336" (UID: "f6011af8-136f-436a-bb13-987d5844f336"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:56:00.786344 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.785331    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-workload-socket" pod "f6011af8-136f-436a-bb13-987d5844f336" (UID: "f6011af8-136f-436a-bb13-987d5844f336"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:56:00.786344 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.785566    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-workload-certs" pod "f6011af8-136f-436a-bb13-987d5844f336" (UID: "f6011af8-136f-436a-bb13-987d5844f336"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:56:00.786344 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.785997    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-credential-socket" pod "f6011af8-136f-436a-bb13-987d5844f336" (UID: "f6011af8-136f-436a-bb13-987d5844f336"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:56:00.787201 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.786589    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-istio-envoy" pod "f6011af8-136f-436a-bb13-987d5844f336" (UID: "f6011af8-136f-436a-bb13-987d5844f336"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:56:00.787874 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.787652    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6011af8-136f-436a-bb13-987d5844f336-kube-api-access-ck2jv" pod "f6011af8-136f-436a-bb13-987d5844f336" (UID: "f6011af8-136f-436a-bb13-987d5844f336"). InnerVolumeSpecName "kube-api-access-ck2jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:56:00.787874 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.787692    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6011af8-136f-436a-bb13-987d5844f336-istio-token" pod "f6011af8-136f-436a-bb13-987d5844f336" (UID: "f6011af8-136f-436a-bb13-987d5844f336"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:56:00.788048 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.787921    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6011af8-136f-436a-bb13-987d5844f336-istio-podinfo" pod "f6011af8-136f-436a-bb13-987d5844f336" (UID: "f6011af8-136f-436a-bb13-987d5844f336"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:56:00.886484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.886291    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck2jv\" (UniqueName: \"kubernetes.io/projected/f6011af8-136f-436a-bb13-987d5844f336-kube-api-access-ck2jv\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:56:00.886484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.886327    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:56:00.886484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.886341    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6011af8-136f-436a-bb13-987d5844f336-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:56:00.886484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.886354    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:56:00.886484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.886367    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:56:00.886484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.886378    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6011af8-136f-436a-bb13-987d5844f336-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:56:00.886484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.886390    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:56:00.886484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.886401    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6011af8-136f-436a-bb13-987d5844f336-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:56:00.886484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.886415    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6011af8-136f-436a-bb13-987d5844f336-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:56:00.888163 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.888111    5650 generic.go:356] "Generic (PLEG): container finished" podID="f6011af8-136f-436a-bb13-987d5844f336" containerID="5308604b5f247586a17fad9cb72a29828c5cab523852a19adae7b777b07f7f6a" exitCode=0
Oct 31 01:56:00.890499 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.888394    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"
Oct 31 01:56:00.891741 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.888415    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z" event={"ID":"f6011af8-136f-436a-bb13-987d5844f336","Type":"ContainerDied","Data":"5308604b5f247586a17fad9cb72a29828c5cab523852a19adae7b777b07f7f6a"}
Oct 31 01:56:00.891857 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.891760    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z" event={"ID":"f6011af8-136f-436a-bb13-987d5844f336","Type":"ContainerDied","Data":"8be05334a3858f48e63eb9c14176d7c37417d5c4915d2bf8d49a393e27e47395"}
Oct 31 01:56:00.891857 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.891797    5650 scope.go:122] "RemoveContainer" containerID="5308604b5f247586a17fad9cb72a29828c5cab523852a19adae7b777b07f7f6a"
Oct 31 01:56:00.964683 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.960637    5650 scope.go:122] "RemoveContainer" containerID="bbba164712e42fa94a9c7cc2065cc488af9e37267c86017d4c457dfc8af744fa"
Oct 31 01:56:00.991817 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.989947    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 31 01:56:00.991817 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.990017    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2" podUID="6aa3fc04-120c-4dd7-ba46-99b133d092b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 31 01:56:01.007477 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:00.994901    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"]
Oct 31 01:56:01.017037 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:01.017000    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-dbff33-openshift-default-6d66d88fb5-6xr8z"]
Oct 31 01:56:01.042693 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:01.026741    5650 scope.go:122] "RemoveContainer" containerID="5308604b5f247586a17fad9cb72a29828c5cab523852a19adae7b777b07f7f6a"
Oct 31 01:56:01.042693 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:56:01.027318    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5308604b5f247586a17fad9cb72a29828c5cab523852a19adae7b777b07f7f6a\": container with ID starting with 5308604b5f247586a17fad9cb72a29828c5cab523852a19adae7b777b07f7f6a not found: ID does not exist" containerID="5308604b5f247586a17fad9cb72a29828c5cab523852a19adae7b777b07f7f6a"
Oct 31 01:56:01.042693 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:01.027355    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5308604b5f247586a17fad9cb72a29828c5cab523852a19adae7b777b07f7f6a"} err="failed to get container status \"5308604b5f247586a17fad9cb72a29828c5cab523852a19adae7b777b07f7f6a\": rpc error: code = NotFound desc = could not find container \"5308604b5f247586a17fad9cb72a29828c5cab523852a19adae7b777b07f7f6a\": container with ID starting with 5308604b5f247586a17fad9cb72a29828c5cab523852a19adae7b777b07f7f6a not found: ID does not exist"
Oct 31 01:56:01.042693 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:01.027384    5650 scope.go:122] "RemoveContainer" containerID="bbba164712e42fa94a9c7cc2065cc488af9e37267c86017d4c457dfc8af744fa"
Oct 31 01:56:01.042693 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:56:01.033998    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbba164712e42fa94a9c7cc2065cc488af9e37267c86017d4c457dfc8af744fa\": container with ID starting with bbba164712e42fa94a9c7cc2065cc488af9e37267c86017d4c457dfc8af744fa not found: ID does not exist" containerID="bbba164712e42fa94a9c7cc2065cc488af9e37267c86017d4c457dfc8af744fa"
Oct 31 01:56:01.042693 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:01.034039    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbba164712e42fa94a9c7cc2065cc488af9e37267c86017d4c457dfc8af744fa"} err="failed to get container status \"bbba164712e42fa94a9c7cc2065cc488af9e37267c86017d4c457dfc8af744fa\": rpc error: code = NotFound desc = could not find container \"bbba164712e42fa94a9c7cc2065cc488af9e37267c86017d4c457dfc8af744fa\": container with ID starting with bbba164712e42fa94a9c7cc2065cc488af9e37267c86017d4c457dfc8af744fa not found: ID does not exist"
Oct 31 01:56:01.939483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:01.939323    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 31 01:56:01.940117 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:01.940080    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2" podUID="6aa3fc04-120c-4dd7-ba46-99b133d092b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 31 01:56:02.313500 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:02.313462    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6011af8-136f-436a-bb13-987d5844f336" path="/var/lib/kubelet/pods/f6011af8-136f-436a-bb13-987d5844f336/volumes"
Oct 31 01:56:03.031977 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:03.031943    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 31 01:56:03.082733 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:03.082688    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2" podUID="6aa3fc04-120c-4dd7-ba46-99b133d092b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 31 01:56:03.615470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:03.614514    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:03.943030 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:03.942430    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 31 01:56:03.943030 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:03.942548    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2" podUID="6aa3fc04-120c-4dd7-ba46-99b133d092b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 31 01:56:04.938877 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:04.938842    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 31 01:56:04.939329 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:04.938905    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2" podUID="6aa3fc04-120c-4dd7-ba46-99b133d092b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 31 01:56:05.939232 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:05.939181    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 31 01:56:05.939641 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:05.939266    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2" podUID="6aa3fc04-120c-4dd7-ba46-99b133d092b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 31 01:56:06.938989 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:06.938638    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 31 01:56:06.938989 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:06.938690    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2" podUID="6aa3fc04-120c-4dd7-ba46-99b133d092b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 31 01:56:07.741482 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:07.741289    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l"
Oct 31 01:56:07.940987 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:07.939252    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 31 01:56:07.940987 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:07.939334    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2" podUID="6aa3fc04-120c-4dd7-ba46-99b133d092b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 31 01:56:08.445279 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:08.445224    5650 patch_prober.go:29] interesting pod/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:56:08.445506 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:08.445317    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8" podUID="fe7e7396-a227-4486-97ba-8b48945cc1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:08.780510 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:08.780403    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx"]
Oct 31 01:56:08.781701 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:08.781666    5650 cpu_manager.go:414] "containerMap: removing container" podUID="f6011af8-136f-436a-bb13-987d5844f336" containerName="istio-proxy"
Oct 31 01:56:08.781701 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:08.781698    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6011af8-136f-436a-bb13-987d5844f336" containerName="istio-proxy"
Oct 31 01:56:08.781912 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:08.781742    5650 cpu_manager.go:414] "containerMap: removing container" podUID="f6011af8-136f-436a-bb13-987d5844f336" containerName="istio-proxy"
Oct 31 01:56:08.781912 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:08.781750    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6011af8-136f-436a-bb13-987d5844f336" containerName="istio-proxy"
Oct 31 01:56:08.782029 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:08.781956    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6011af8-136f-436a-bb13-987d5844f336" containerName="istio-proxy"
Oct 31 01:56:08.852573 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:08.852282    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx"]
Oct 31 01:56:08.852573 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:08.852340    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"]
Oct 31 01:56:08.867952 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:08.859806    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx"
Oct 31 01:56:08.867952 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:08.861936    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6011af8-136f-436a-bb13-987d5844f336" containerName="istio-proxy"
Oct 31 01:56:08.876569 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:08.870083    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-c8b702\""
Oct 31 01:56:08.877658 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:08.877624    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"]
Oct 31 01:56:08.877950 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:08.877929    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj"]
Oct 31 01:56:08.879155 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:08.878240    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:08.891847 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:08.891810    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-e57add-openshift-default-dockercfg-qzwh9\""
Oct 31 01:56:08.905949 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:08.903700    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv7tb\" (UniqueName: \"kubernetes.io/projected/07980c5c-538d-4f86-b518-a1a03d86a8f9-kube-api-access-gv7tb\") pod \"payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx\" (UID: \"07980c5c-538d-4f86-b518-a1a03d86a8f9\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx"
Oct 31 01:56:08.905949 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:08.903892    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07980c5c-538d-4f86-b518-a1a03d86a8f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx\" (UID: \"07980c5c-538d-4f86-b518-a1a03d86a8f9\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx"
Oct 31 01:56:09.005461 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.005401    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-token\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.006082 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.006057    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-credential-socket\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.006266 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.006250    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv7tb\" (UniqueName: \"kubernetes.io/projected/07980c5c-538d-4f86-b518-a1a03d86a8f9-kube-api-access-gv7tb\") pod \"payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx\" (UID: \"07980c5c-538d-4f86-b518-a1a03d86a8f9\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx"
Oct 31 01:56:09.006487 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.006470    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-podinfo\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.006630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.006613    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-data\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.006734 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.006720    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-envoy\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.006841 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.006826    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaaaa634-58c5-4a1f-9eee-816fe2256844-istiod-ca-cert\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.006976 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.006961    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07980c5c-538d-4f86-b518-a1a03d86a8f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx\" (UID: \"07980c5c-538d-4f86-b518-a1a03d86a8f9\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx"
Oct 31 01:56:09.007143 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.007128    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-workload-certs\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.007252 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.007237    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7bln\" (UniqueName: \"kubernetes.io/projected/eaaaa634-58c5-4a1f-9eee-816fe2256844-kube-api-access-m7bln\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.007359 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.007346    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-workload-socket\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.009055 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.009028    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07980c5c-538d-4f86-b518-a1a03d86a8f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx\" (UID: \"07980c5c-538d-4f86-b518-a1a03d86a8f9\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx"
Oct 31 01:56:09.068357 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.068273    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj"]
Oct 31 01:56:09.098124 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.078350    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj"
Oct 31 01:56:09.110299 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.110264    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwcvk\" (UniqueName: \"kubernetes.io/projected/a2fd40dc-d829-4940-87c3-bd54c6559f84-kube-api-access-bwcvk\") pod \"payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj\" (UID: \"a2fd40dc-d829-4940-87c3-bd54c6559f84\") " pod="openshift-ingress/payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj"
Oct 31 01:56:09.110616 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.110574    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-token\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.110872 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.110848    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2fd40dc-d829-4940-87c3-bd54c6559f84-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj\" (UID: \"a2fd40dc-d829-4940-87c3-bd54c6559f84\") " pod="openshift-ingress/payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj"
Oct 31 01:56:09.111068 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.111050    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-credential-socket\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.111346 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.111324    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-podinfo\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.111543 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.111512    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-data\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.111710 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.111693    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-envoy\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.111855 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.111838    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaaaa634-58c5-4a1f-9eee-816fe2256844-istiod-ca-cert\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.113370 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.113350    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-workload-certs\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.113600 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.113581    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7bln\" (UniqueName: \"kubernetes.io/projected/eaaaa634-58c5-4a1f-9eee-816fe2256844-kube-api-access-m7bln\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.113749 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.113734    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-workload-socket\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.192593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.112430    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-c8b702-dockercfg-66hd9\""
Oct 31 01:56:09.192593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.113061    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaaaa634-58c5-4a1f-9eee-816fe2256844-istiod-ca-cert\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.192593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.158765    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-credential-socket\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.192593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.159400    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv7tb\" (UniqueName: \"kubernetes.io/projected/07980c5c-538d-4f86-b518-a1a03d86a8f9-kube-api-access-gv7tb\") pod \"payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx\" (UID: \"07980c5c-538d-4f86-b518-a1a03d86a8f9\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx"
Oct 31 01:56:09.192593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.160324    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-workload-certs\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.192593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.160542    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-data\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.192593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.160675    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-workload-socket\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.192593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.168384    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-podinfo\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.192593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.168978    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-envoy\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.192593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.175175    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-token\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.192593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.175284    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7bln\" (UniqueName: \"kubernetes.io/projected/eaaaa634-58c5-4a1f-9eee-816fe2256844-kube-api-access-m7bln\") pod \"e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") " pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.214412 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.207158    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:09.259299 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.255705    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2fd40dc-d829-4940-87c3-bd54c6559f84-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj\" (UID: \"a2fd40dc-d829-4940-87c3-bd54c6559f84\") " pod="openshift-ingress/payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj"
Oct 31 01:56:09.259299 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.256295    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwcvk\" (UniqueName: \"kubernetes.io/projected/a2fd40dc-d829-4940-87c3-bd54c6559f84-kube-api-access-bwcvk\") pod \"payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj\" (UID: \"a2fd40dc-d829-4940-87c3-bd54c6559f84\") " pod="openshift-ingress/payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj"
Oct 31 01:56:09.259299 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.257666    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2fd40dc-d829-4940-87c3-bd54c6559f84-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj\" (UID: \"a2fd40dc-d829-4940-87c3-bd54c6559f84\") " pod="openshift-ingress/payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj"
Oct 31 01:56:09.272375 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.272052    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwcvk\" (UniqueName: \"kubernetes.io/projected/a2fd40dc-d829-4940-87c3-bd54c6559f84-kube-api-access-bwcvk\") pod \"payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj\" (UID: \"a2fd40dc-d829-4940-87c3-bd54c6559f84\") " pod="openshift-ingress/payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj"
Oct 31 01:56:09.285558 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.285255    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj"
Oct 31 01:56:09.383718 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.383332    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx"
Oct 31 01:56:09.653363 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.653310    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"]
Oct 31 01:56:09.653902 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:56:09.653867    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaaaa634_58c5_4a1f_9eee_816fe2256844.slice/crio-fc692c02a1929e6540ea63d48e986ab04eb998239a52f375d77961ffd732bf99 WatchSource:0}: Error finding container fc692c02a1929e6540ea63d48e986ab04eb998239a52f375d77961ffd732bf99: Status 404 returned error can't find the container with id fc692c02a1929e6540ea63d48e986ab04eb998239a52f375d77961ffd732bf99
Oct 31 01:56:09.658035 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.657985    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:56:09.672543 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.658152    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:56:09.672543 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.658242    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:56:09.717723 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:56:09.717693    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2fd40dc_d829_4940_87c3_bd54c6559f84.slice/crio-11002b9038e1c64fb027b4fb623d08408a9e6d9258f140fdbb29d37b7af8538a WatchSource:0}: Error finding container 11002b9038e1c64fb027b4fb623d08408a9e6d9258f140fdbb29d37b7af8538a: Status 404 returned error can't find the container with id 11002b9038e1c64fb027b4fb623d08408a9e6d9258f140fdbb29d37b7af8538a
Oct 31 01:56:09.725704 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.725671    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:56:09.728710 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.727757    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:56:09.733645 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.733619    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 01:56:09.735060 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.735033    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj"]
Oct 31 01:56:09.854412 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:09.854323    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx"]
Oct 31 01:56:10.245269 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:10.245229    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx" event={"ID":"07980c5c-538d-4f86-b518-a1a03d86a8f9","Type":"ContainerStarted","Data":"d4a7813c5ada3a629e81a1f13f4d17498dc2dd59fe73b6e5f9ab256263b97620"}
Oct 31 01:56:10.245497 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:10.245295    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx" event={"ID":"07980c5c-538d-4f86-b518-a1a03d86a8f9","Type":"ContainerStarted","Data":"8e6dd7423faaaf4af91685ae78db741ea9b4d058073773d68fa5a9c099977b5e"}
Oct 31 01:56:10.245497 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:10.245326    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx"
Oct 31 01:56:10.247414 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:10.247375    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj" event={"ID":"a2fd40dc-d829-4940-87c3-bd54c6559f84","Type":"ContainerStarted","Data":"a80c4a3939ec818e1e72eb4c72d410e896f77ece8b7ff18d08bc8b243ab6dcaa"}
Oct 31 01:56:10.247634 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:10.247432    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj" event={"ID":"a2fd40dc-d829-4940-87c3-bd54c6559f84","Type":"ContainerStarted","Data":"11002b9038e1c64fb027b4fb623d08408a9e6d9258f140fdbb29d37b7af8538a"}
Oct 31 01:56:10.247634 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:10.247557    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj"
Oct 31 01:56:10.250572 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:10.250543    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb" event={"ID":"eaaaa634-58c5-4a1f-9eee-816fe2256844","Type":"ContainerStarted","Data":"0806c99cef4632b374470b8fdb2f40f55371bcc91dace685ed0094a0ddfa2857"}
Oct 31 01:56:10.250703 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:10.250592    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb" event={"ID":"eaaaa634-58c5-4a1f-9eee-816fe2256844","Type":"ContainerStarted","Data":"fc692c02a1929e6540ea63d48e986ab04eb998239a52f375d77961ffd732bf99"}
Oct 31 01:56:10.272152 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:10.272086    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx" podStartSLOduration=2.272068541 podStartE2EDuration="2.272068541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:10.266101324 +0000 UTC m=+2126.269815616" watchObservedRunningTime="2026-10-31 01:56:10.272068541 +0000 UTC m=+2126.275782917"
Oct 31 01:56:10.291953 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:10.291875    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj" podStartSLOduration=2.291853037 podStartE2EDuration="2.291853037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:10.289280878 +0000 UTC m=+2126.292995169" watchObservedRunningTime="2026-10-31 01:56:10.291853037 +0000 UTC m=+2126.295567331"
Oct 31 01:56:10.345650 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:10.345587    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb" podStartSLOduration=2.345569162 podStartE2EDuration="2.345569162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:10.341681287 +0000 UTC m=+2126.345395595" watchObservedRunningTime="2026-10-31 01:56:10.345569162 +0000 UTC m=+2126.349283454"
Oct 31 01:56:11.218328 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:11.218297    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:11.257561 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:11.224086    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:11.263421 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:11.263396    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:11.264665 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:11.264632    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:56:12.281027 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:12.280994    5650 generic.go:356] "Generic (PLEG): container finished" podID="247b6d6e-faaf-44e3-bf73-73fbb620f5e0" containerID="c7c398c7b60890635bba29a9eb4f236353288f7452fc2dc48736905d4bb563b5" exitCode=0
Oct 31 01:56:12.281571 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:12.281060    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8" event={"ID":"247b6d6e-faaf-44e3-bf73-73fbb620f5e0","Type":"ContainerDied","Data":"c7c398c7b60890635bba29a9eb4f236353288f7452fc2dc48736905d4bb563b5"}
Oct 31 01:56:12.281671 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:12.281651    5650 scope.go:122] "RemoveContainer" containerID="c7c398c7b60890635bba29a9eb4f236353288f7452fc2dc48736905d4bb563b5"
Oct 31 01:56:12.283095 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:12.283073    5650 generic.go:356] "Generic (PLEG): container finished" podID="fe7e7396-a227-4486-97ba-8b48945cc1d3" containerID="273ebfeeb08e9008b5e71fc27415a534e938057c3de950849f7161c7bc6f14d8" exitCode=0
Oct 31 01:56:12.283237 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:12.283147    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8" event={"ID":"fe7e7396-a227-4486-97ba-8b48945cc1d3","Type":"ContainerDied","Data":"273ebfeeb08e9008b5e71fc27415a534e938057c3de950849f7161c7bc6f14d8"}
Oct 31 01:56:12.283615 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:12.283582    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:56:12.283682 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:12.283662    5650 scope.go:122] "RemoveContainer" containerID="273ebfeeb08e9008b5e71fc27415a534e938057c3de950849f7161c7bc6f14d8"
Oct 31 01:56:12.283760 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:12.283664    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:56:12.283807 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:12.283793    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:56:12.285473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:12.285431    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:56:12.285565 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:12.285511    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:56:12.285565 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:12.285546    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:56:13.294053 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.294020    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8" event={"ID":"247b6d6e-faaf-44e3-bf73-73fbb620f5e0","Type":"ContainerStarted","Data":"0763e56a16c51dd04345c4a2ab527e5d949a68d26a69689278349dcd65b672c3"}
Oct 31 01:56:13.297312 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.297281    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8" event={"ID":"fe7e7396-a227-4486-97ba-8b48945cc1d3","Type":"ContainerStarted","Data":"ada616721ed81bea6d5fcf304f4443e6241ffaa0ff2dc5d4d8a16783a9f13d59"}
Oct 31 01:56:13.498122 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.498085    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"]
Oct 31 01:56:13.518211 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.512789    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.548531 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.546984    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:56:13.552469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.551331    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 31 01:56:13.552469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.551386    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8" podUID="247b6d6e-faaf-44e3-bf73-73fbb620f5e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 31 01:56:13.574853 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.574799    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"]
Oct 31 01:56:13.624471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.624226    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e09af575-0369-4421-9326-9836c5910c67-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.624471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.624299    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.624471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.624333    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09af575-0369-4421-9326-9836c5910c67-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.624471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.624370    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.624471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.624414    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.624471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.624464    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.624967 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.624630    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.624967 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.624733    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slx9z\" (UniqueName: \"kubernetes.io/projected/e09af575-0369-4421-9326-9836c5910c67-kube-api-access-slx9z\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.624967 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.624858    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09af575-0369-4421-9326-9836c5910c67-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.750469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.749574    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09af575-0369-4421-9326-9836c5910c67-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.750469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.749673    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e09af575-0369-4421-9326-9836c5910c67-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.750469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.749754    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.750469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.749792    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09af575-0369-4421-9326-9836c5910c67-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.750469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.749835    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.750469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.749879    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.750469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.749918    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.750469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.750081    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.750469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.750182    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slx9z\" (UniqueName: \"kubernetes.io/projected/e09af575-0369-4421-9326-9836c5910c67-kube-api-access-slx9z\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.754469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.751406    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09af575-0369-4421-9326-9836c5910c67-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.757527 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.756066    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.758491 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.758225    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.758633 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.758557    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.758834 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.758808    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.759000 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.758950    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.759382 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.759359    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09af575-0369-4421-9326-9836c5910c67-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.763870 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.763846    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e09af575-0369-4421-9326-9836c5910c67-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.764907 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.764887    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slx9z\" (UniqueName: \"kubernetes.io/projected/e09af575-0369-4421-9326-9836c5910c67-kube-api-access-slx9z\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:13.894487 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:13.894384    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:14.067808 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:14.067767    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:56:14.067983 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:14.067873    5650 patch_prober.go:29] interesting pod/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 01:56:14.067983 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:14.067923    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8" podUID="fe7e7396-a227-4486-97ba-8b48945cc1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 01:56:14.079388 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:14.075201    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv"]
Oct 31 01:56:14.084235 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:14.083001    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv"
Oct 31 01:56:14.124341 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:14.104079    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-e57add-dockercfg-fdnfw\""
Oct 31 01:56:14.124341 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:14.124121    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-e57add\""
Oct 31 01:56:14.163937 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:14.163569    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv"]
Oct 31 01:56:14.423530 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:14.420207    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrk87\" (UniqueName: \"kubernetes.io/projected/938e1ac7-e7cd-4773-bce5-305fbf793670-kube-api-access-jrk87\") pod \"payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv\" (UID: \"938e1ac7-e7cd-4773-bce5-305fbf793670\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv"
Oct 31 01:56:14.423530 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:14.420746    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/938e1ac7-e7cd-4773-bce5-305fbf793670-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv\" (UID: \"938e1ac7-e7cd-4773-bce5-305fbf793670\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv"
Oct 31 01:56:14.554477 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:14.549926    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 31 01:56:14.557618 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:14.557570    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8" podUID="247b6d6e-faaf-44e3-bf73-73fbb620f5e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 31 01:56:14.586127 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:14.586091    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/938e1ac7-e7cd-4773-bce5-305fbf793670-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv\" (UID: \"938e1ac7-e7cd-4773-bce5-305fbf793670\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv"
Oct 31 01:56:14.586636 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:14.586610    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrk87\" (UniqueName: \"kubernetes.io/projected/938e1ac7-e7cd-4773-bce5-305fbf793670-kube-api-access-jrk87\") pod \"payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv\" (UID: \"938e1ac7-e7cd-4773-bce5-305fbf793670\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv"
Oct 31 01:56:14.588189 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:14.588165    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/938e1ac7-e7cd-4773-bce5-305fbf793670-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv\" (UID: \"938e1ac7-e7cd-4773-bce5-305fbf793670\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv"
Oct 31 01:56:14.677753 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:14.677677    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrk87\" (UniqueName: \"kubernetes.io/projected/938e1ac7-e7cd-4773-bce5-305fbf793670-kube-api-access-jrk87\") pod \"payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv\" (UID: \"938e1ac7-e7cd-4773-bce5-305fbf793670\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv"
Oct 31 01:56:14.790502 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:14.790462    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv"
Oct 31 01:56:14.887925 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:14.883936    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"]
Oct 31 01:56:14.990868 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:14.990811    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:56:14.991083 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:14.991060    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:56:14.991219 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:14.991203    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:56:15.059528 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:15.054633    5650 patch_prober.go:29] interesting pod/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 01:56:15.059528 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:15.054742    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8" podUID="fe7e7396-a227-4486-97ba-8b48945cc1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 01:56:15.370807 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:15.369266    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv"]
Oct 31 01:56:15.555551 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:15.554427    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 31 01:56:15.555551 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:15.554545    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8" podUID="247b6d6e-faaf-44e3-bf73-73fbb620f5e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 31 01:56:15.859760 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:15.859631    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv" event={"ID":"938e1ac7-e7cd-4773-bce5-305fbf793670","Type":"ContainerStarted","Data":"27706eb2c1596b41369eb5469a64913c9fa3cfabd27d00d8e664e13f4c1711d1"}
Oct 31 01:56:15.863802 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:15.862255    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72" event={"ID":"e09af575-0369-4421-9326-9836c5910c67","Type":"ContainerStarted","Data":"4260e90a47600f3ef95eb71d4cbaae8cc0efc25a68787cfc41b3b33a766efd93"}
Oct 31 01:56:15.863802 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:15.862315    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72" event={"ID":"e09af575-0369-4421-9326-9836c5910c67","Type":"ContainerStarted","Data":"52cd8667ba64ecdccc59c985906f3c60d531019bb908658275a08a73c278a576"}
Oct 31 01:56:15.901767 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:15.896521    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72" podStartSLOduration=2.8964999540000003 podStartE2EDuration="2.896499954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:15.894941017 +0000 UTC m=+2131.898655308" watchObservedRunningTime="2026-10-31 01:56:15.896499954 +0000 UTC m=+2131.900214247"
Oct 31 01:56:15.920494 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:15.914906    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:16.052989 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:16.052552    5650 patch_prober.go:29] interesting pod/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 01:56:16.052989 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:16.052607    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8" podUID="fe7e7396-a227-4486-97ba-8b48945cc1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 01:56:16.596552 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:16.566076    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 31 01:56:16.596552 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:16.590666    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8" podUID="247b6d6e-faaf-44e3-bf73-73fbb620f5e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 31 01:56:16.897914 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:16.897458    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 31 01:56:16.897914 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:16.897569    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72" podUID="e09af575-0369-4421-9326-9836c5910c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 31 01:56:16.910389 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:16.910255    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv" event={"ID":"938e1ac7-e7cd-4773-bce5-305fbf793670","Type":"ContainerStarted","Data":"4d3ff82861e3b26d77c95b919373cbd5629ab6eaadd02ab4839cee050f1005b3"}
Oct 31 01:56:16.910389 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:16.910348    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv"
Oct 31 01:56:16.944258 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:16.944177    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv" podStartSLOduration=2.944153524 podStartE2EDuration="2.944153524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:16.934928614 +0000 UTC m=+2132.938642905" watchObservedRunningTime="2026-10-31 01:56:16.944153524 +0000 UTC m=+2132.947867820"
Oct 31 01:56:17.052033 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:17.051996    5650 patch_prober.go:29] interesting pod/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 01:56:17.052352 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:17.052303    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8" podUID="fe7e7396-a227-4486-97ba-8b48945cc1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 01:56:17.529470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:17.520561    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:17.554543 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:17.546991    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 31 01:56:17.554543 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:17.547055    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8" podUID="247b6d6e-faaf-44e3-bf73-73fbb620f5e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 31 01:56:17.598473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:17.598120    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:17.671473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:17.669168    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:17.671473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:17.669638    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 01:56:17.671473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:17.671169    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:56:17.671825 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:17.671511    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:17.672027 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:17.671996    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:17.898536 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:17.895058    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 31 01:56:17.917412 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:17.917359    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72" podUID="e09af575-0369-4421-9326-9836c5910c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 31 01:56:18.095842 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:18.095805    5650 patch_prober.go:29] interesting pod/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 01:56:18.096178 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:18.096132    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8" podUID="fe7e7396-a227-4486-97ba-8b48945cc1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 01:56:18.161525 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:18.161102    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded"
Oct 31 01:56:18.424968 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:18.424839    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:18.442754 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:18.442710    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:18.552938 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:18.552898    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 31 01:56:18.554393 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:18.554346    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8" podUID="247b6d6e-faaf-44e3-bf73-73fbb620f5e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 31 01:56:18.572860 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:18.572816    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:18.574048 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:18.574021    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:56:18.929473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:18.904787    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 31 01:56:18.929473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:18.920573    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72" podUID="e09af575-0369-4421-9326-9836c5910c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 31 01:56:18.929473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:18.925405    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 01:56:19.087472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:19.087412    5650 patch_prober.go:29] interesting pod/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 01:56:19.094283 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:19.094237    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8" podUID="fe7e7396-a227-4486-97ba-8b48945cc1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 01:56:19.438472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:19.437532    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:19.438472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:19.438347    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:56:19.540889 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:19.540838    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 01:56:19.586580 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:19.586107    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Oct 31 01:56:19.587569 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:19.587020    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 01:56:19.594232 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:19.593507    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 31 01:56:19.594232 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:19.593569    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8" podUID="247b6d6e-faaf-44e3-bf73-73fbb620f5e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 31 01:56:19.635270 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:19.628819    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:19.897138 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:19.896995    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 31 01:56:19.897138 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:19.897056    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72" podUID="e09af575-0369-4421-9326-9836c5910c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 31 01:56:20.059146 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:20.059110    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:56:20.059650 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:20.059586    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:56:20.061143 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:20.061104    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 01:56:20.553878 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:20.553828    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:56:20.555690 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:20.555666    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 01:56:20.895464 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:20.895354    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 31 01:56:20.895464 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:20.895419    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72" podUID="e09af575-0369-4421-9326-9836c5910c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 31 01:56:21.267937 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:21.267892    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj"
Oct 31 01:56:21.269673 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:21.269650    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx"
Oct 31 01:56:21.896557 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:21.896518    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 31 01:56:21.896857 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:21.896820    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72" podUID="e09af575-0369-4421-9326-9836c5910c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 31 01:56:22.897473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:22.897413    5650 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 31 01:56:22.898121 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:22.898073    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72" podUID="e09af575-0369-4421-9326-9836c5910c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 31 01:56:23.917629 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:23.916982    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:23.917629 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:23.917567    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:23.918571 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:23.918543    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 01:56:28.764604 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:56:28.764561    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:56:29.239340 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:29.239311    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv"
Oct 31 01:56:34.479530 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.479488    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"]
Oct 31 01:56:34.491117 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.491088    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 01:56:34.498287 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.498254    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"]
Oct 31 01:56:34.498756 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.498728    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-7a850e\"/\"auth-test-model-7a850e-kserve-self-signed-certs\""
Oct 31 01:56:34.498860 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.498846    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-7a850e\"/\"default-dockercfg-bd7xz\""
Oct 31 01:56:34.563416 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.562573    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3aa84cad-c52d-4da2-bb1f-622bf9738e23-tls-certs\") pod \"auth-test-model-7a850e-kserve-6565d6b7df-prgdx\" (UID: \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\") " pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 01:56:34.563416 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.562748    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2sqk\" (UniqueName: \"kubernetes.io/projected/3aa84cad-c52d-4da2-bb1f-622bf9738e23-kube-api-access-p2sqk\") pod \"auth-test-model-7a850e-kserve-6565d6b7df-prgdx\" (UID: \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\") " pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 01:56:34.563416 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.562857    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-home\") pod \"auth-test-model-7a850e-kserve-6565d6b7df-prgdx\" (UID: \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\") " pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 01:56:34.563416 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.562896    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-tmp-dir\") pod \"auth-test-model-7a850e-kserve-6565d6b7df-prgdx\" (UID: \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\") " pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 01:56:34.563416 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.562937    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-model-cache\") pod \"auth-test-model-7a850e-kserve-6565d6b7df-prgdx\" (UID: \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\") " pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 01:56:34.563416 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.563039    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-dshm\") pod \"auth-test-model-7a850e-kserve-6565d6b7df-prgdx\" (UID: \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\") " pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 01:56:34.564296 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.564135    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-7a850e\"/\"kube-root-ca.crt\""
Oct 31 01:56:34.564296 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.564253    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-7a850e\"/\"openshift-service-ca.crt\""
Oct 31 01:56:34.696868 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.696831    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-dshm\") pod \"auth-test-model-7a850e-kserve-6565d6b7df-prgdx\" (UID: \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\") " pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 01:56:34.697073 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.696933    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3aa84cad-c52d-4da2-bb1f-622bf9738e23-tls-certs\") pod \"auth-test-model-7a850e-kserve-6565d6b7df-prgdx\" (UID: \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\") " pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 01:56:34.697073 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.697040    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2sqk\" (UniqueName: \"kubernetes.io/projected/3aa84cad-c52d-4da2-bb1f-622bf9738e23-kube-api-access-p2sqk\") pod \"auth-test-model-7a850e-kserve-6565d6b7df-prgdx\" (UID: \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\") " pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 01:56:34.697192 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.697127    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-home\") pod \"auth-test-model-7a850e-kserve-6565d6b7df-prgdx\" (UID: \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\") " pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 01:56:34.697192 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.697161    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-tmp-dir\") pod \"auth-test-model-7a850e-kserve-6565d6b7df-prgdx\" (UID: \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\") " pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 01:56:34.697297 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.697197    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-model-cache\") pod \"auth-test-model-7a850e-kserve-6565d6b7df-prgdx\" (UID: \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\") " pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 01:56:34.697724 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.697702    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-model-cache\") pod \"auth-test-model-7a850e-kserve-6565d6b7df-prgdx\" (UID: \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\") " pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 01:56:34.698464 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.698423    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-tmp-dir\") pod \"auth-test-model-7a850e-kserve-6565d6b7df-prgdx\" (UID: \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\") " pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 01:56:34.698602 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.698544    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-home\") pod \"auth-test-model-7a850e-kserve-6565d6b7df-prgdx\" (UID: \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\") " pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 01:56:34.719127 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.719089    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3aa84cad-c52d-4da2-bb1f-622bf9738e23-tls-certs\") pod \"auth-test-model-7a850e-kserve-6565d6b7df-prgdx\" (UID: \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\") " pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 01:56:34.736550 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.733821    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-dshm\") pod \"auth-test-model-7a850e-kserve-6565d6b7df-prgdx\" (UID: \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\") " pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 01:56:34.744464 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.738868    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2sqk\" (UniqueName: \"kubernetes.io/projected/3aa84cad-c52d-4da2-bb1f-622bf9738e23-kube-api-access-p2sqk\") pod \"auth-test-model-7a850e-kserve-6565d6b7df-prgdx\" (UID: \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\") " pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 01:56:34.866885 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:34.866845    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 01:56:35.257552 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:35.250018    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"]
Oct 31 01:56:35.266345 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:56:35.266313    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3aa84cad_c52d_4da2_bb1f_622bf9738e23.slice/crio-17dbbead1f52ba250dc549596c206a53f69e418e91f967bb81ef9e973e93390c WatchSource:0}: Error finding container 17dbbead1f52ba250dc549596c206a53f69e418e91f967bb81ef9e973e93390c: Status 404 returned error can't find the container with id 17dbbead1f52ba250dc549596c206a53f69e418e91f967bb81ef9e973e93390c
Oct 31 01:56:35.689880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:35.689781    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx" event={"ID":"3aa84cad-c52d-4da2-bb1f-622bf9738e23","Type":"ContainerStarted","Data":"17ed1c83938cb18306a59f3720c9ad9a8d583a70ffa05eb9d6f6516510c2ea82"}
Oct 31 01:56:35.689880 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:35.689833    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx" event={"ID":"3aa84cad-c52d-4da2-bb1f-622bf9738e23","Type":"ContainerStarted","Data":"17dbbead1f52ba250dc549596c206a53f69e418e91f967bb81ef9e973e93390c"}
Oct 31 01:56:35.728439 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:35.715590    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx" podStartSLOduration=1.715568754 podStartE2EDuration="1.715568754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:35.713069212 +0000 UTC m=+2151.716783507" watchObservedRunningTime="2026-10-31 01:56:35.715568754 +0000 UTC m=+2151.719283048"
Oct 31 01:56:35.867376 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:35.867333    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 01:56:39.714599 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:39.714556    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:56:39.715092 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:39.714625    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf" podUID="f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:42.774173 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:42.774078    5650 generic.go:356] "Generic (PLEG): container finished" podID="0d39c057-51e1-49f1-85de-facd9e8f15fc" containerID="59a60e4b44b4c5a3fb2fafb89a0f33421e9c07de1142c2dd75c40fb56d727ddd" exitCode=0
Oct 31 01:56:42.774765 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:42.774154    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np" event={"ID":"0d39c057-51e1-49f1-85de-facd9e8f15fc","Type":"ContainerDied","Data":"59a60e4b44b4c5a3fb2fafb89a0f33421e9c07de1142c2dd75c40fb56d727ddd"}
Oct 31 01:56:42.774832 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:42.774777    5650 scope.go:122] "RemoveContainer" containerID="59a60e4b44b4c5a3fb2fafb89a0f33421e9c07de1142c2dd75c40fb56d727ddd"
Oct 31 01:56:42.776738 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:42.776713    5650 generic.go:356] "Generic (PLEG): container finished" podID="f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" containerID="a40a5f0f3953f38da44794fc7ab5fce6c1efea4f791b8205452a6b6d054beb6d" exitCode=0
Oct 31 01:56:42.776874 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:42.776792    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:56:42.776924 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:42.776874    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf" event={"ID":"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db","Type":"ContainerDied","Data":"a40a5f0f3953f38da44794fc7ab5fce6c1efea4f791b8205452a6b6d054beb6d"}
Oct 31 01:56:42.776924 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:42.776879    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:56:42.776991 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:42.776926    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:56:42.777286 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:42.777270    5650 scope.go:122] "RemoveContainer" containerID="a40a5f0f3953f38da44794fc7ab5fce6c1efea4f791b8205452a6b6d054beb6d"
Oct 31 01:56:42.779072 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:42.779044    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:56:42.779190 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:42.779108    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:56:42.779190 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:42.779142    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:56:43.791387 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:43.791351    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np" event={"ID":"0d39c057-51e1-49f1-85de-facd9e8f15fc","Type":"ContainerStarted","Data":"0f3f7de50e68596f8b21df5e920197215208339ecb95644d9fa3cd80501526e0"}
Oct 31 01:56:43.793521 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:43.793498    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf" event={"ID":"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db","Type":"ContainerStarted","Data":"5a429146af346981ba9fcf8cc978728f4cddc165d49836dc3cba6c0131a4ae8e"}
Oct 31 01:56:43.991192 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:43.991128    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:56:43.991519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:43.991498    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 31 01:56:43.991585 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:43.991550    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf" podUID="f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 31 01:56:44.345500 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:44.342791    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:56:44.366652 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:44.366618    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 31 01:56:44.366898 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:44.366876    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np" podUID="0d39c057-51e1-49f1-85de-facd9e8f15fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 31 01:56:44.867307 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:44.867270    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 01:56:44.880045 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:44.880021    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 01:56:44.991704 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:44.991664    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 31 01:56:44.991889 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:44.991738    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf" podUID="f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 31 01:56:45.347903 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:45.347866    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 31 01:56:45.349665 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:45.349085    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np" podUID="0d39c057-51e1-49f1-85de-facd9e8f15fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 31 01:56:45.529624 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:45.529574    5650 scope.go:122] "RemoveContainer" containerID="b20a32d2e3ba1e811ca3980dd61da593c80d447a59be8844f4acf2bd7e9c0021"
Oct 31 01:56:45.860999 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:45.860888    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 01:56:46.182581 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:46.182091    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 31 01:56:46.182581 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:46.182149    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf" podUID="f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 31 01:56:46.342505 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:46.339118    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 31 01:56:46.342505 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:46.339199    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np" podUID="0d39c057-51e1-49f1-85de-facd9e8f15fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 31 01:56:47.022717 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:47.022675    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 31 01:56:47.023039 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:47.023005    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf" podUID="f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 31 01:56:47.341047 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:47.340947    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 31 01:56:47.341747 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:47.341701    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np" podUID="0d39c057-51e1-49f1-85de-facd9e8f15fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 31 01:56:47.547608 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:47.547554    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 01:56:47.556588 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:47.556550    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:56:47.593348 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:47.592219    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 01:56:47.625959 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:47.625459    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 01:56:47.631480 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:47.625381    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:47.636600 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:47.636548    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:47.842592 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:47.842537    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx" podUID="3aa84cad-c52d-4da2-bb1f-622bf9738e23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.176:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:47.969530 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:47.954588    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:47.992396 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:47.992353    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 31 01:56:47.992913 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:47.992875    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf" podUID="f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 31 01:56:48.151762 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:48.151718    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:48.346532 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:48.338021    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 31 01:56:48.346532 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:48.338114    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np" podUID="0d39c057-51e1-49f1-85de-facd9e8f15fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 31 01:56:49.123778 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.123738    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"]
Oct 31 01:56:49.237583 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.225143    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"
Oct 31 01:56:49.427605 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.248990    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/462d1f98-0942-4bed-8654-41b0bc4725b0-tls-certs\") pod \"explicit-model-c8b702-kserve-54494d5895-vzv8j\" (UID: \"462d1f98-0942-4bed-8654-41b0bc4725b0\") " pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"
Oct 31 01:56:49.427605 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.407596    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-model-cache\") pod \"explicit-model-c8b702-kserve-54494d5895-vzv8j\" (UID: \"462d1f98-0942-4bed-8654-41b0bc4725b0\") " pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"
Oct 31 01:56:49.427605 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.407757    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-dshm\") pod \"explicit-model-c8b702-kserve-54494d5895-vzv8j\" (UID: \"462d1f98-0942-4bed-8654-41b0bc4725b0\") " pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"
Oct 31 01:56:49.427605 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.408074    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-tmp-dir\") pod \"explicit-model-c8b702-kserve-54494d5895-vzv8j\" (UID: \"462d1f98-0942-4bed-8654-41b0bc4725b0\") " pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"
Oct 31 01:56:49.427605 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.408147    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75v5n\" (UniqueName: \"kubernetes.io/projected/462d1f98-0942-4bed-8654-41b0bc4725b0-kube-api-access-75v5n\") pod \"explicit-model-c8b702-kserve-54494d5895-vzv8j\" (UID: \"462d1f98-0942-4bed-8654-41b0bc4725b0\") " pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"
Oct 31 01:56:49.427605 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.408188    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-home\") pod \"explicit-model-c8b702-kserve-54494d5895-vzv8j\" (UID: \"462d1f98-0942-4bed-8654-41b0bc4725b0\") " pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"
Oct 31 01:56:49.435791 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.435756    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c8b702\"/\"openshift-service-ca.crt\""
Oct 31 01:56:49.436210 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.436187    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c8b702\"/\"default-dockercfg-g5hhr\""
Oct 31 01:56:49.436505 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.436488    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c8b702\"/\"explicit-model-c8b702-kserve-self-signed-certs\""
Oct 31 01:56:49.436690 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.436674    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c8b702\"/\"kube-root-ca.crt\""
Oct 31 01:56:49.480069 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.356825    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"]
Oct 31 01:56:49.480496 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.480469    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 31 01:56:49.480715 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.480690    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np" podUID="0d39c057-51e1-49f1-85de-facd9e8f15fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 31 01:56:49.533473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.514696    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-dshm\") pod \"explicit-model-c8b702-kserve-54494d5895-vzv8j\" (UID: \"462d1f98-0942-4bed-8654-41b0bc4725b0\") " pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"
Oct 31 01:56:49.533473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.514935    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-tmp-dir\") pod \"explicit-model-c8b702-kserve-54494d5895-vzv8j\" (UID: \"462d1f98-0942-4bed-8654-41b0bc4725b0\") " pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"
Oct 31 01:56:49.533473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.514998    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75v5n\" (UniqueName: \"kubernetes.io/projected/462d1f98-0942-4bed-8654-41b0bc4725b0-kube-api-access-75v5n\") pod \"explicit-model-c8b702-kserve-54494d5895-vzv8j\" (UID: \"462d1f98-0942-4bed-8654-41b0bc4725b0\") " pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"
Oct 31 01:56:49.533473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.515035    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-home\") pod \"explicit-model-c8b702-kserve-54494d5895-vzv8j\" (UID: \"462d1f98-0942-4bed-8654-41b0bc4725b0\") " pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"
Oct 31 01:56:49.533473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.515206    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/462d1f98-0942-4bed-8654-41b0bc4725b0-tls-certs\") pod \"explicit-model-c8b702-kserve-54494d5895-vzv8j\" (UID: \"462d1f98-0942-4bed-8654-41b0bc4725b0\") " pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"
Oct 31 01:56:49.533473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.515262    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-model-cache\") pod \"explicit-model-c8b702-kserve-54494d5895-vzv8j\" (UID: \"462d1f98-0942-4bed-8654-41b0bc4725b0\") " pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"
Oct 31 01:56:49.533473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.515901    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-model-cache\") pod \"explicit-model-c8b702-kserve-54494d5895-vzv8j\" (UID: \"462d1f98-0942-4bed-8654-41b0bc4725b0\") " pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"
Oct 31 01:56:49.533473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.525715    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-tmp-dir\") pod \"explicit-model-c8b702-kserve-54494d5895-vzv8j\" (UID: \"462d1f98-0942-4bed-8654-41b0bc4725b0\") " pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"
Oct 31 01:56:49.533473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.526241    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-home\") pod \"explicit-model-c8b702-kserve-54494d5895-vzv8j\" (UID: \"462d1f98-0942-4bed-8654-41b0bc4725b0\") " pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"
Oct 31 01:56:49.557469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.551653    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 31 01:56:49.557469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.551727    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf" podUID="f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 31 01:56:49.678578 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.667708    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/462d1f98-0942-4bed-8654-41b0bc4725b0-tls-certs\") pod \"explicit-model-c8b702-kserve-54494d5895-vzv8j\" (UID: \"462d1f98-0942-4bed-8654-41b0bc4725b0\") " pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"
Oct 31 01:56:49.713832 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.712249    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75v5n\" (UniqueName: \"kubernetes.io/projected/462d1f98-0942-4bed-8654-41b0bc4725b0-kube-api-access-75v5n\") pod \"explicit-model-c8b702-kserve-54494d5895-vzv8j\" (UID: \"462d1f98-0942-4bed-8654-41b0bc4725b0\") " pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"
Oct 31 01:56:49.760205 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.758845    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-dshm\") pod \"explicit-model-c8b702-kserve-54494d5895-vzv8j\" (UID: \"462d1f98-0942-4bed-8654-41b0bc4725b0\") " pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"
Oct 31 01:56:49.866610 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.866262    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"
Oct 31 01:56:49.994501 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.994436    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 31 01:56:49.995187 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:49.995149    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf" podUID="f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 31 01:56:50.365093 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:50.340032    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Oct 31 01:56:50.365093 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:50.340093    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np" podUID="0d39c057-51e1-49f1-85de-facd9e8f15fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Oct 31 01:56:50.563611 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:50.563508    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"]
Oct 31 01:56:50.572781 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:56:50.572653    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod462d1f98_0942_4bed_8654_41b0bc4725b0.slice/crio-dcf37556662a98339a2ee365fbbb80f3c51994a5db391c7f37909b1cf5b79663 WatchSource:0}: Error finding container dcf37556662a98339a2ee365fbbb80f3c51994a5db391c7f37909b1cf5b79663: Status 404 returned error can't find the container with id dcf37556662a98339a2ee365fbbb80f3c51994a5db391c7f37909b1cf5b79663
Oct 31 01:56:50.998200 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:50.998069    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:56:50.998575 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:50.998556    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:56:51.013518 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:50.999520    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 01:56:51.355357 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:51.355277    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:56:51.357217 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:51.357190    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 01:56:51.566869 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:51.566824    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j" event={"ID":"462d1f98-0942-4bed-8654-41b0bc4725b0","Type":"ContainerStarted","Data":"6087aaf13c0d5a649335c2bfcd95658e915d4169baf96330359eb1b7ad43b083"}
Oct 31 01:56:51.567320 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:51.566881    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j" event={"ID":"462d1f98-0942-4bed-8654-41b0bc4725b0","Type":"ContainerStarted","Data":"dcf37556662a98339a2ee365fbbb80f3c51994a5db391c7f37909b1cf5b79663"}
Oct 31 01:56:51.593102 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:51.593050    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j" podStartSLOduration=3.593032334 podStartE2EDuration="3.593032334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:51.590659779 +0000 UTC m=+2167.594374070" watchObservedRunningTime="2026-10-31 01:56:51.593032334 +0000 UTC m=+2167.596746622"
Oct 31 01:56:51.869510 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:51.868284    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"
Oct 31 01:56:52.933523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:52.933486    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"]
Oct 31 01:56:52.941334 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:52.941200    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"]
Oct 31 01:56:52.941852 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:52.941775    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:56:52.948537 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:52.947459    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-6d7f5d\"/\"auth-test-model-6d7f5d-kserve-self-signed-certs\""
Oct 31 01:56:52.948537 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:52.948501    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-6d7f5d\"/\"openshift-service-ca.crt\""
Oct 31 01:56:52.950285 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:52.950261    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-6d7f5d\"/\"kube-root-ca.crt\""
Oct 31 01:56:52.950634 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:52.950538    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-6d7f5d\"/\"default-dockercfg-zbbw5\""
Oct 31 01:56:53.024614 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:53.024564    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-tmp-dir\") pod \"auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr\" (UID: \"b3da8c75-360a-438e-9394-0e78ed5eb558\") " pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:56:53.024947 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:53.024927    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-home\") pod \"auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr\" (UID: \"b3da8c75-360a-438e-9394-0e78ed5eb558\") " pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:56:53.025077 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:53.025054    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-dshm\") pod \"auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr\" (UID: \"b3da8c75-360a-438e-9394-0e78ed5eb558\") " pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:56:53.025372 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:53.025351    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3da8c75-360a-438e-9394-0e78ed5eb558-tls-certs\") pod \"auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr\" (UID: \"b3da8c75-360a-438e-9394-0e78ed5eb558\") " pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:56:53.025653 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:53.025632    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-model-cache\") pod \"auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr\" (UID: \"b3da8c75-360a-438e-9394-0e78ed5eb558\") " pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:56:53.025829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:53.025809    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8hdh\" (UniqueName: \"kubernetes.io/projected/b3da8c75-360a-438e-9394-0e78ed5eb558-kube-api-access-s8hdh\") pod \"auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr\" (UID: \"b3da8c75-360a-438e-9394-0e78ed5eb558\") " pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:56:53.143663 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:53.143613    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3da8c75-360a-438e-9394-0e78ed5eb558-tls-certs\") pod \"auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr\" (UID: \"b3da8c75-360a-438e-9394-0e78ed5eb558\") " pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:56:53.144571 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:53.144543    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-model-cache\") pod \"auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr\" (UID: \"b3da8c75-360a-438e-9394-0e78ed5eb558\") " pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:56:53.145339 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:53.145202    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-model-cache\") pod \"auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr\" (UID: \"b3da8c75-360a-438e-9394-0e78ed5eb558\") " pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:56:53.145595 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:53.145572    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8hdh\" (UniqueName: \"kubernetes.io/projected/b3da8c75-360a-438e-9394-0e78ed5eb558-kube-api-access-s8hdh\") pod \"auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr\" (UID: \"b3da8c75-360a-438e-9394-0e78ed5eb558\") " pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:56:53.148426 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:53.148379    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-tmp-dir\") pod \"auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr\" (UID: \"b3da8c75-360a-438e-9394-0e78ed5eb558\") " pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:56:53.150513 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:53.150489    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-tmp-dir\") pod \"auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr\" (UID: \"b3da8c75-360a-438e-9394-0e78ed5eb558\") " pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:56:53.150832 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:53.150809    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-home\") pod \"auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr\" (UID: \"b3da8c75-360a-438e-9394-0e78ed5eb558\") " pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:56:53.151641 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:53.151484    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-home\") pod \"auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr\" (UID: \"b3da8c75-360a-438e-9394-0e78ed5eb558\") " pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:56:53.152138 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:53.152119    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-dshm\") pod \"auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr\" (UID: \"b3da8c75-360a-438e-9394-0e78ed5eb558\") " pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:56:53.185774 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:53.158595    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3da8c75-360a-438e-9394-0e78ed5eb558-tls-certs\") pod \"auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr\" (UID: \"b3da8c75-360a-438e-9394-0e78ed5eb558\") " pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:56:53.185774 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:53.182639    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-dshm\") pod \"auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr\" (UID: \"b3da8c75-360a-438e-9394-0e78ed5eb558\") " pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:56:53.186008 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:53.185904    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8hdh\" (UniqueName: \"kubernetes.io/projected/b3da8c75-360a-438e-9394-0e78ed5eb558-kube-api-access-s8hdh\") pod \"auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr\" (UID: \"b3da8c75-360a-438e-9394-0e78ed5eb558\") " pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:56:53.262384 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:53.260269    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:56:53.710685 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:56:53.710654    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3da8c75_360a_438e_9394_0e78ed5eb558.slice/crio-8fbb65d5a777c5ac6b675976c5e7c708fd9255acae0f241ebbb3d455a83755c8 WatchSource:0}: Error finding container 8fbb65d5a777c5ac6b675976c5e7c708fd9255acae0f241ebbb3d455a83755c8: Status 404 returned error can't find the container with id 8fbb65d5a777c5ac6b675976c5e7c708fd9255acae0f241ebbb3d455a83755c8
Oct 31 01:56:53.711350 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:53.711327    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"]
Oct 31 01:56:54.603801 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:54.603760    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr" event={"ID":"b3da8c75-360a-438e-9394-0e78ed5eb558","Type":"ContainerStarted","Data":"300dd3cbd99d40f47ea905b9c13af5511599b20fc2a36d856685d119c4a25d0a"}
Oct 31 01:56:54.604296 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:54.603814    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr" event={"ID":"b3da8c75-360a-438e-9394-0e78ed5eb558","Type":"ContainerStarted","Data":"8fbb65d5a777c5ac6b675976c5e7c708fd9255acae0f241ebbb3d455a83755c8"}
Oct 31 01:56:54.628226 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:54.628169    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr" podStartSLOduration=2.6281554160000002 podStartE2EDuration="2.628155416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:54.624149065 +0000 UTC m=+2170.627863357" watchObservedRunningTime="2026-10-31 01:56:54.628155416 +0000 UTC m=+2170.631869706"
Oct 31 01:56:55.260829 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:55.260790    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:56:56.621552 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:56.620521    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"]
Oct 31 01:56:57.304624 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:57.304581    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:56:57.304849 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:57.304655    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb" podUID="eaaaa634-58c5-4a1f-9eee-816fe2256844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:58.554505 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:56:58.554408    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:56:58.655515 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:58.655480    5650 generic.go:356] "Generic (PLEG): container finished" podID="eaaaa634-58c5-4a1f-9eee-816fe2256844" containerID="0806c99cef4632b374470b8fdb2f40f55371bcc91dace685ed0094a0ddfa2857" exitCode=0
Oct 31 01:56:58.655687 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:58.655555    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb" event={"ID":"eaaaa634-58c5-4a1f-9eee-816fe2256844","Type":"ContainerDied","Data":"0806c99cef4632b374470b8fdb2f40f55371bcc91dace685ed0094a0ddfa2857"}
Oct 31 01:56:58.656065 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:58.656046    5650 scope.go:122] "RemoveContainer" containerID="0806c99cef4632b374470b8fdb2f40f55371bcc91dace685ed0094a0ddfa2857"
Oct 31 01:56:58.657973 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:58.657942    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:56:58.658049 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:58.658015    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:56:58.658086 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:58.658062    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:56:59.671478 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:56:59.670921    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb" event={"ID":"eaaaa634-58c5-4a1f-9eee-816fe2256844","Type":"ContainerStarted","Data":"f91d7ad3a82a89dffedda938f8a2ef057ff0abbf7423959f3f7748eae1b12a6d"}
Oct 31 01:57:00.232467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:00.231991    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:57:00.232467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:00.232108    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 01:57:00.232467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:00.232151    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb" podUID="eaaaa634-58c5-4a1f-9eee-816fe2256844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 01:57:01.221470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:01.221157    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 01:57:01.221470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:01.221223    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb" podUID="eaaaa634-58c5-4a1f-9eee-816fe2256844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 01:57:02.228466 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:02.223543    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 01:57:02.228466 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:02.223612    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb" podUID="eaaaa634-58c5-4a1f-9eee-816fe2256844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 01:57:03.228554 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:03.227127    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 01:57:03.228554 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:03.227196    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb" podUID="eaaaa634-58c5-4a1f-9eee-816fe2256844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 01:57:03.261335 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:03.261300    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:57:03.274160 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:03.274131    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:57:03.757053 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:03.757019    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:57:04.225386 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:04.225343    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 01:57:04.225639 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:04.225420    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb" podUID="eaaaa634-58c5-4a1f-9eee-816fe2256844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 01:57:05.250471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:05.249496    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 01:57:05.250471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:05.249577    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb" podUID="eaaaa634-58c5-4a1f-9eee-816fe2256844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 01:57:06.223149 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:06.223118    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:57:06.223430 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:06.223409    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:57:06.224322 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:06.224303    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 01:57:07.495431 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:07.495387    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj"]
Oct 31 01:57:07.513088 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:07.513039    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx"]
Oct 31 01:57:11.675179 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:11.675113    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j" podUID="462d1f98-0942-4bed-8654-41b0bc4725b0" containerName="main" containerID="cri-o://6087aaf13c0d5a649335c2bfcd95658e915d4169baf96330359eb1b7ad43b083" gracePeriod=45
Oct 31 01:57:11.831654 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:11.831619    5650 generic.go:356] "Generic (PLEG): container finished" podID="462d1f98-0942-4bed-8654-41b0bc4725b0" containerID="6087aaf13c0d5a649335c2bfcd95658e915d4169baf96330359eb1b7ad43b083" exitCode=0
Oct 31 01:57:11.831845 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:11.831665    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j" event={"ID":"462d1f98-0942-4bed-8654-41b0bc4725b0","Type":"ContainerDied","Data":"6087aaf13c0d5a649335c2bfcd95658e915d4169baf96330359eb1b7ad43b083"}
Oct 31 01:57:12.035938 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.035894    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"
Oct 31 01:57:12.082653 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.082615    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-model-cache\") pod \"462d1f98-0942-4bed-8654-41b0bc4725b0\" (UID: \"462d1f98-0942-4bed-8654-41b0bc4725b0\") "
Oct 31 01:57:12.082842 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.082716    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/462d1f98-0942-4bed-8654-41b0bc4725b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/462d1f98-0942-4bed-8654-41b0bc4725b0-tls-certs\") pod \"462d1f98-0942-4bed-8654-41b0bc4725b0\" (UID: \"462d1f98-0942-4bed-8654-41b0bc4725b0\") "
Oct 31 01:57:12.082842 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.082834    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-tmp-dir\") pod \"462d1f98-0942-4bed-8654-41b0bc4725b0\" (UID: \"462d1f98-0942-4bed-8654-41b0bc4725b0\") "
Oct 31 01:57:12.082999 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.082970    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-home\" (UniqueName: \"kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-home\") pod \"462d1f98-0942-4bed-8654-41b0bc4725b0\" (UID: \"462d1f98-0942-4bed-8654-41b0bc4725b0\") "
Oct 31 01:57:12.083081 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.082960    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-model-cache" pod "462d1f98-0942-4bed-8654-41b0bc4725b0" (UID: "462d1f98-0942-4bed-8654-41b0bc4725b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:12.083322 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.083134    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/462d1f98-0942-4bed-8654-41b0bc4725b0-kube-api-access-75v5n\" (UniqueName: \"kubernetes.io/projected/462d1f98-0942-4bed-8654-41b0bc4725b0-kube-api-access-75v5n\") pod \"462d1f98-0942-4bed-8654-41b0bc4725b0\" (UID: \"462d1f98-0942-4bed-8654-41b0bc4725b0\") "
Oct 31 01:57:12.083322 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.083162    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-tmp-dir" pod "462d1f98-0942-4bed-8654-41b0bc4725b0" (UID: "462d1f98-0942-4bed-8654-41b0bc4725b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:12.083322 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.083300    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-dshm\") pod \"462d1f98-0942-4bed-8654-41b0bc4725b0\" (UID: \"462d1f98-0942-4bed-8654-41b0bc4725b0\") "
Oct 31 01:57:12.083847 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.083826    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-home" pod "462d1f98-0942-4bed-8654-41b0bc4725b0" (UID: "462d1f98-0942-4bed-8654-41b0bc4725b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:12.084020 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.083998    5650 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-tmp-dir\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:12.084100 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.084024    5650 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-model-cache\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:12.085861 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.085837    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462d1f98-0942-4bed-8654-41b0bc4725b0-kube-api-access-75v5n" pod "462d1f98-0942-4bed-8654-41b0bc4725b0" (UID: "462d1f98-0942-4bed-8654-41b0bc4725b0"). InnerVolumeSpecName "kube-api-access-75v5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:57:12.086539 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.086503    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-dshm" pod "462d1f98-0942-4bed-8654-41b0bc4725b0" (UID: "462d1f98-0942-4bed-8654-41b0bc4725b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:12.086651 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.086567    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/462d1f98-0942-4bed-8654-41b0bc4725b0-tls-certs" pod "462d1f98-0942-4bed-8654-41b0bc4725b0" (UID: "462d1f98-0942-4bed-8654-41b0bc4725b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:57:12.185826 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.185779    5650 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-dshm\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:12.185826 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.185821    5650 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/462d1f98-0942-4bed-8654-41b0bc4725b0-tls-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:12.185826 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.185836    5650 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/462d1f98-0942-4bed-8654-41b0bc4725b0-home\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:12.186082 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.185850    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75v5n\" (UniqueName: \"kubernetes.io/projected/462d1f98-0942-4bed-8654-41b0bc4725b0-kube-api-access-75v5n\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:12.496515 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.496475    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj" podUID="a2fd40dc-d829-4940-87c3-bd54c6559f84" containerName="payload-processing" containerID="cri-o://a80c4a3939ec818e1e72eb4c72d410e896f77ece8b7ff18d08bc8b243ab6dcaa" gracePeriod=25
Oct 31 01:57:12.514039 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.513983    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx" podUID="07980c5c-538d-4f86-b518-a1a03d86a8f9" containerName="payload-pre-processing" containerID="cri-o://d4a7813c5ada3a629e81a1f13f4d17498dc2dd59fe73b6e5f9ab256263b97620" gracePeriod=25
Oct 31 01:57:12.849198 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.849160    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j" event={"ID":"462d1f98-0942-4bed-8654-41b0bc4725b0","Type":"ContainerDied","Data":"dcf37556662a98339a2ee365fbbb80f3c51994a5db391c7f37909b1cf5b79663"}
Oct 31 01:57:12.849651 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.849226    5650 scope.go:122] "RemoveContainer" containerID="6087aaf13c0d5a649335c2bfcd95658e915d4169baf96330359eb1b7ad43b083"
Oct 31 01:57:12.849651 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.849405    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"
Oct 31 01:57:12.863510 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.863216    5650 generic.go:356] "Generic (PLEG): container finished" podID="07980c5c-538d-4f86-b518-a1a03d86a8f9" containerID="d4a7813c5ada3a629e81a1f13f4d17498dc2dd59fe73b6e5f9ab256263b97620" exitCode=0
Oct 31 01:57:12.863510 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.863285    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx" event={"ID":"07980c5c-538d-4f86-b518-a1a03d86a8f9","Type":"ContainerDied","Data":"d4a7813c5ada3a629e81a1f13f4d17498dc2dd59fe73b6e5f9ab256263b97620"}
Oct 31 01:57:12.865426 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.865394    5650 generic.go:356] "Generic (PLEG): container finished" podID="a2fd40dc-d829-4940-87c3-bd54c6559f84" containerID="a80c4a3939ec818e1e72eb4c72d410e896f77ece8b7ff18d08bc8b243ab6dcaa" exitCode=0
Oct 31 01:57:12.865595 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.865518    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj" event={"ID":"a2fd40dc-d829-4940-87c3-bd54c6559f84","Type":"ContainerDied","Data":"a80c4a3939ec818e1e72eb4c72d410e896f77ece8b7ff18d08bc8b243ab6dcaa"}
Oct 31 01:57:12.865595 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.865577    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj" event={"ID":"a2fd40dc-d829-4940-87c3-bd54c6559f84","Type":"ContainerDied","Data":"11002b9038e1c64fb027b4fb623d08408a9e6d9258f140fdbb29d37b7af8538a"}
Oct 31 01:57:12.865692 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.865600    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11002b9038e1c64fb027b4fb623d08408a9e6d9258f140fdbb29d37b7af8538a"
Oct 31 01:57:12.869109 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.869021    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj"
Oct 31 01:57:12.907352 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.906320    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2fd40dc-d829-4940-87c3-bd54c6559f84-kube-api-access-bwcvk\" (UniqueName: \"kubernetes.io/projected/a2fd40dc-d829-4940-87c3-bd54c6559f84-kube-api-access-bwcvk\") pod \"a2fd40dc-d829-4940-87c3-bd54c6559f84\" (UID: \"a2fd40dc-d829-4940-87c3-bd54c6559f84\") "
Oct 31 01:57:12.907352 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.906543    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2fd40dc-d829-4940-87c3-bd54c6559f84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2fd40dc-d829-4940-87c3-bd54c6559f84-plugins-config-volume\") pod \"a2fd40dc-d829-4940-87c3-bd54c6559f84\" (UID: \"a2fd40dc-d829-4940-87c3-bd54c6559f84\") "
Oct 31 01:57:12.908189 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.907855    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2fd40dc-d829-4940-87c3-bd54c6559f84-plugins-config-volume" pod "a2fd40dc-d829-4940-87c3-bd54c6559f84" (UID: "a2fd40dc-d829-4940-87c3-bd54c6559f84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:57:12.910170 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.910145    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2fd40dc-d829-4940-87c3-bd54c6559f84-kube-api-access-bwcvk" pod "a2fd40dc-d829-4940-87c3-bd54c6559f84" (UID: "a2fd40dc-d829-4940-87c3-bd54c6559f84"). InnerVolumeSpecName "kube-api-access-bwcvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:57:12.912934 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.912906    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"]
Oct 31 01:57:12.919963 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.919936    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-c8b702/explicit-model-c8b702-kserve-54494d5895-vzv8j"]
Oct 31 01:57:12.952576 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:12.952545    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx"
Oct 31 01:57:13.007603 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.007552    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07980c5c-538d-4f86-b518-a1a03d86a8f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07980c5c-538d-4f86-b518-a1a03d86a8f9-plugins-config-volume\") pod \"07980c5c-538d-4f86-b518-a1a03d86a8f9\" (UID: \"07980c5c-538d-4f86-b518-a1a03d86a8f9\") "
Oct 31 01:57:13.007872 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.007853    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07980c5c-538d-4f86-b518-a1a03d86a8f9-kube-api-access-gv7tb\" (UniqueName: \"kubernetes.io/projected/07980c5c-538d-4f86-b518-a1a03d86a8f9-kube-api-access-gv7tb\") pod \"07980c5c-538d-4f86-b518-a1a03d86a8f9\" (UID: \"07980c5c-538d-4f86-b518-a1a03d86a8f9\") "
Oct 31 01:57:13.008114 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.008083    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07980c5c-538d-4f86-b518-a1a03d86a8f9-plugins-config-volume" pod "07980c5c-538d-4f86-b518-a1a03d86a8f9" (UID: "07980c5c-538d-4f86-b518-a1a03d86a8f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:57:13.008615 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.008588    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2fd40dc-d829-4940-87c3-bd54c6559f84-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:13.008721 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.008619    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07980c5c-538d-4f86-b518-a1a03d86a8f9-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:13.008721 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.008635    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwcvk\" (UniqueName: \"kubernetes.io/projected/a2fd40dc-d829-4940-87c3-bd54c6559f84-kube-api-access-bwcvk\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:13.010180 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.010156    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07980c5c-538d-4f86-b518-a1a03d86a8f9-kube-api-access-gv7tb" pod "07980c5c-538d-4f86-b518-a1a03d86a8f9" (UID: "07980c5c-538d-4f86-b518-a1a03d86a8f9"). InnerVolumeSpecName "kube-api-access-gv7tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:57:13.110087 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.109985    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv7tb\" (UniqueName: \"kubernetes.io/projected/07980c5c-538d-4f86-b518-a1a03d86a8f9-kube-api-access-gv7tb\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:13.666026 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.665991    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"]
Oct 31 01:57:13.666832 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.666807    5650 cpu_manager.go:414] "containerMap: removing container" podUID="462d1f98-0942-4bed-8654-41b0bc4725b0" containerName="main"
Oct 31 01:57:13.666832 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.666828    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="462d1f98-0942-4bed-8654-41b0bc4725b0" containerName="main"
Oct 31 01:57:13.667020 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.666850    5650 cpu_manager.go:414] "containerMap: removing container" podUID="07980c5c-538d-4f86-b518-a1a03d86a8f9" containerName="payload-pre-processing"
Oct 31 01:57:13.667020 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.666856    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07980c5c-538d-4f86-b518-a1a03d86a8f9" containerName="payload-pre-processing"
Oct 31 01:57:13.667020 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.666898    5650 cpu_manager.go:414] "containerMap: removing container" podUID="a2fd40dc-d829-4940-87c3-bd54c6559f84" containerName="payload-processing"
Oct 31 01:57:13.667020 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.666904    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2fd40dc-d829-4940-87c3-bd54c6559f84" containerName="payload-processing"
Oct 31 01:57:13.667020 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.667015    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2fd40dc-d829-4940-87c3-bd54c6559f84" containerName="payload-processing"
Oct 31 01:57:13.667294 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.667027    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="462d1f98-0942-4bed-8654-41b0bc4725b0" containerName="main"
Oct 31 01:57:13.667294 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.667037    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="07980c5c-538d-4f86-b518-a1a03d86a8f9" containerName="payload-pre-processing"
Oct 31 01:57:13.671242 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.671213    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:13.678674 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.678483    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-e57add\"/\"openshift-service-ca.crt\""
Oct 31 01:57:13.679956 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.679921    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-e57add\"/\"kube-root-ca.crt\""
Oct 31 01:57:13.681225 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.681178    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"]
Oct 31 01:57:13.691512 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.691481    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-e57add\"/\"default-dockercfg-kc5p5\""
Oct 31 01:57:13.691659 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.691637    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-e57add\"/\"auth-test-model-e57add-kserve-self-signed-certs\""
Oct 31 01:57:13.717912 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.717873    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0221d5f2-b6f6-4aaf-950a-b1028238ee91-tls-certs\") pod \"auth-test-model-e57add-kserve-79d85df557-tpr4m\" (UID: \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\") " pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:13.718194 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.717938    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-home\") pod \"auth-test-model-e57add-kserve-79d85df557-tpr4m\" (UID: \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\") " pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:13.718194 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.717990    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-dshm\") pod \"auth-test-model-e57add-kserve-79d85df557-tpr4m\" (UID: \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\") " pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:13.718194 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.718135    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhbj2\" (UniqueName: \"kubernetes.io/projected/0221d5f2-b6f6-4aaf-950a-b1028238ee91-kube-api-access-hhbj2\") pod \"auth-test-model-e57add-kserve-79d85df557-tpr4m\" (UID: \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\") " pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:13.718548 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.718221    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-tmp-dir\") pod \"auth-test-model-e57add-kserve-79d85df557-tpr4m\" (UID: \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\") " pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:13.718548 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.718276    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-model-cache\") pod \"auth-test-model-e57add-kserve-79d85df557-tpr4m\" (UID: \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\") " pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:13.819964 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.819918    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-model-cache\") pod \"auth-test-model-e57add-kserve-79d85df557-tpr4m\" (UID: \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\") " pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:13.820212 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.820056    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0221d5f2-b6f6-4aaf-950a-b1028238ee91-tls-certs\") pod \"auth-test-model-e57add-kserve-79d85df557-tpr4m\" (UID: \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\") " pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:13.820212 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.820109    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-home\") pod \"auth-test-model-e57add-kserve-79d85df557-tpr4m\" (UID: \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\") " pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:13.820212 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.820173    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-dshm\") pod \"auth-test-model-e57add-kserve-79d85df557-tpr4m\" (UID: \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\") " pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:13.820486 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.820341    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhbj2\" (UniqueName: \"kubernetes.io/projected/0221d5f2-b6f6-4aaf-950a-b1028238ee91-kube-api-access-hhbj2\") pod \"auth-test-model-e57add-kserve-79d85df557-tpr4m\" (UID: \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\") " pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:13.820486 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.820370    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-model-cache\") pod \"auth-test-model-e57add-kserve-79d85df557-tpr4m\" (UID: \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\") " pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:13.820643 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.820489    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-tmp-dir\") pod \"auth-test-model-e57add-kserve-79d85df557-tpr4m\" (UID: \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\") " pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:13.820703 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.820683    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-home\") pod \"auth-test-model-e57add-kserve-79d85df557-tpr4m\" (UID: \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\") " pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:13.820988 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.820964    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-tmp-dir\") pod \"auth-test-model-e57add-kserve-79d85df557-tpr4m\" (UID: \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\") " pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:13.850475 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.845794    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-dshm\") pod \"auth-test-model-e57add-kserve-79d85df557-tpr4m\" (UID: \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\") " pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:13.871734 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.865927    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0221d5f2-b6f6-4aaf-950a-b1028238ee91-tls-certs\") pod \"auth-test-model-e57add-kserve-79d85df557-tpr4m\" (UID: \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\") " pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:13.976463 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:13.952821    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhbj2\" (UniqueName: \"kubernetes.io/projected/0221d5f2-b6f6-4aaf-950a-b1028238ee91-kube-api-access-hhbj2\") pod \"auth-test-model-e57add-kserve-79d85df557-tpr4m\" (UID: \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\") " pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:14.007262 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:14.007197    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx" event={"ID":"07980c5c-538d-4f86-b518-a1a03d86a8f9","Type":"ContainerDied","Data":"8e6dd7423faaaf4af91685ae78db741ea9b4d058073773d68fa5a9c099977b5e"}
Oct 31 01:57:14.007611 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:14.007288    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx"
Oct 31 01:57:14.008042 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:14.008018    5650 scope.go:122] "RemoveContainer" containerID="d4a7813c5ada3a629e81a1f13f4d17498dc2dd59fe73b6e5f9ab256263b97620"
Oct 31 01:57:14.014106 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:14.012885    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:14.014690 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:14.014659    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj"
Oct 31 01:57:14.063225 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:14.063182    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj"]
Oct 31 01:57:14.066751 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:14.066723    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c8b702-85569ff95c-rvksj"]
Oct 31 01:57:14.081297 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:14.081210    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx"]
Oct 31 01:57:14.087288 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:14.087257    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c8b702-6874d9474c-rwvbx"]
Oct 31 01:57:14.133353 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:14.132428    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07980c5c-538d-4f86-b518-a1a03d86a8f9" path="/var/lib/kubelet/pods/07980c5c-538d-4f86-b518-a1a03d86a8f9/volumes"
Oct 31 01:57:14.133353 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:14.133277    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="462d1f98-0942-4bed-8654-41b0bc4725b0" path="/var/lib/kubelet/pods/462d1f98-0942-4bed-8654-41b0bc4725b0/volumes"
Oct 31 01:57:14.133844 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:14.133822    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2fd40dc-d829-4940-87c3-bd54c6559f84" path="/var/lib/kubelet/pods/a2fd40dc-d829-4940-87c3-bd54c6559f84/volumes"
Oct 31 01:57:14.334557 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:57:14.334522    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0221d5f2_b6f6_4aaf_950a_b1028238ee91.slice/crio-9e9d5e3af1028dba035d2de13abbc619aac761c3e645e93cabe17e714ca8549a WatchSource:0}: Error finding container 9e9d5e3af1028dba035d2de13abbc619aac761c3e645e93cabe17e714ca8549a: Status 404 returned error can't find the container with id 9e9d5e3af1028dba035d2de13abbc619aac761c3e645e93cabe17e714ca8549a
Oct 31 01:57:14.334882 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:14.334864    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"]
Oct 31 01:57:15.024774 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:15.024739    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m" event={"ID":"0221d5f2-b6f6-4aaf-950a-b1028238ee91","Type":"ContainerStarted","Data":"748038ee3c580013c9bcff0841ec21512163838b1af2f0a46521a3fbda880288"}
Oct 31 01:57:15.025190 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:15.024786    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m" event={"ID":"0221d5f2-b6f6-4aaf-950a-b1028238ee91","Type":"ContainerStarted","Data":"9e9d5e3af1028dba035d2de13abbc619aac761c3e645e93cabe17e714ca8549a"}
Oct 31 01:57:15.048660 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:15.048605    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m" podStartSLOduration=2.048589656 podStartE2EDuration="2.048589656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:57:15.04547096 +0000 UTC m=+2191.049185253" watchObservedRunningTime="2026-10-31 01:57:15.048589656 +0000 UTC m=+2191.052303946"
Oct 31 01:57:16.013340 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:16.013295    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:24.013734 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:24.013693    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:24.026648 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:24.026614    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:24.120132 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:24.120103    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:27.604105 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.604064    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"]
Oct 31 01:57:27.609936 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.609908    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.618941 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.618501    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-ccfa21-openshift-default-dockercfg-hfpbt\""
Oct 31 01:57:27.630504 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.630433    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"]
Oct 31 01:57:27.680002 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.679960    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-data\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.680229 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.680016    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istiod-ca-cert\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.680229 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.680207    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-credential-socket\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.680415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.680399    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjbz5\" (UniqueName: \"kubernetes.io/projected/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-kube-api-access-vjbz5\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.680507 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.680488    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-workload-socket\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.680550 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.680522    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-workload-certs\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.680591 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.680555    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-envoy\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.680630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.680595    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-podinfo\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.680630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.680612    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-token\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.786469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.784029    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-envoy\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.786469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.784118    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-podinfo\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.786469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.784148    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-token\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.786469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.784258    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-data\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.786469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.784305    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istiod-ca-cert\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.786469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.784399    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-credential-socket\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.787087 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.786583    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istiod-ca-cert\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.787087 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.786855    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-credential-socket\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.787087 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.787041    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-data\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.787350 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.787324    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjbz5\" (UniqueName: \"kubernetes.io/projected/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-kube-api-access-vjbz5\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.787419 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.787401    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-workload-socket\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.787513 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.787434    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-workload-certs\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.791467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.787828    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-workload-certs\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.791467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.788378    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-workload-socket\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.822475 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.806956    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-podinfo\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.822475 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.822037    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-envoy\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.836875 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.828082    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-token\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.851474 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.846057    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjbz5\" (UniqueName: \"kubernetes.io/projected/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-kube-api-access-vjbz5\") pod \"e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") " pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:27.957528 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:27.952766    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:28.161861 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:28.161828    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"]
Oct 31 01:57:28.445906 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:28.445849    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"]
Oct 31 01:57:28.450981 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:57:28.450945    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c0b4f9_f1b1_45e8_b076_ba5768b15a67.slice/crio-26a2aafc798a2bf35537e18b79f720c1716f2681f3fc1dd1279db952bf7091a1 WatchSource:0}: Error finding container 26a2aafc798a2bf35537e18b79f720c1716f2681f3fc1dd1279db952bf7091a1: Status 404 returned error can't find the container with id 26a2aafc798a2bf35537e18b79f720c1716f2681f3fc1dd1279db952bf7091a1
Oct 31 01:57:28.453655 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:28.453618    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:57:28.453753 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:28.453697    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:57:28.453753 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:28.453741    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:57:28.561526 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:57:28.561483    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:57:29.158125 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:29.158087    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p" event={"ID":"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67","Type":"ContainerStarted","Data":"cca43b0d2b44ec9c45e4aecfb2d212a6d3c808d5787184610d517980d1e4d06a"}
Oct 31 01:57:29.158603 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:29.158144    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p" event={"ID":"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67","Type":"ContainerStarted","Data":"26a2aafc798a2bf35537e18b79f720c1716f2681f3fc1dd1279db952bf7091a1"}
Oct 31 01:57:29.186127 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:29.186067    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p" podStartSLOduration=2.186052384 podStartE2EDuration="2.186052384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:57:29.182823963 +0000 UTC m=+2205.186538255" watchObservedRunningTime="2026-10-31 01:57:29.186052384 +0000 UTC m=+2205.189766674"
Oct 31 01:57:29.993654 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:29.993603    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:30.006350 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:30.006317    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:30.171101 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:30.170207    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:30.172609 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:30.172582    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:31.462941 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:31.458584    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"]
Oct 31 01:57:43.216470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:43.195713    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr" podUID="b3da8c75-360a-438e-9394-0e78ed5eb558" containerName="main" containerID="cri-o://300dd3cbd99d40f47ea905b9c13af5511599b20fc2a36d856685d119c4a25d0a" gracePeriod=45
Oct 31 01:57:43.435658 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:43.435594    5650 generic.go:356] "Generic (PLEG): container finished" podID="b3da8c75-360a-438e-9394-0e78ed5eb558" containerID="300dd3cbd99d40f47ea905b9c13af5511599b20fc2a36d856685d119c4a25d0a" exitCode=0
Oct 31 01:57:43.436318 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:43.436291    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr" event={"ID":"b3da8c75-360a-438e-9394-0e78ed5eb558","Type":"ContainerDied","Data":"300dd3cbd99d40f47ea905b9c13af5511599b20fc2a36d856685d119c4a25d0a"}
Oct 31 01:57:43.515107 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:57:43.514997    5650 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3da8c75_360a_438e_9394_0e78ed5eb558.slice/crio-conmon-300dd3cbd99d40f47ea905b9c13af5511599b20fc2a36d856685d119c4a25d0a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3da8c75_360a_438e_9394_0e78ed5eb558.slice/crio-300dd3cbd99d40f47ea905b9c13af5511599b20fc2a36d856685d119c4a25d0a.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 01:57:43.667050 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:43.667017    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:57:43.784521 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:43.784413    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-dshm\") pod \"b3da8c75-360a-438e-9394-0e78ed5eb558\" (UID: \"b3da8c75-360a-438e-9394-0e78ed5eb558\") "
Oct 31 01:57:43.784521 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:43.784490    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-model-cache\") pod \"b3da8c75-360a-438e-9394-0e78ed5eb558\" (UID: \"b3da8c75-360a-438e-9394-0e78ed5eb558\") "
Oct 31 01:57:43.784777 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:43.784575    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3da8c75-360a-438e-9394-0e78ed5eb558-kube-api-access-s8hdh\" (UniqueName: \"kubernetes.io/projected/b3da8c75-360a-438e-9394-0e78ed5eb558-kube-api-access-s8hdh\") pod \"b3da8c75-360a-438e-9394-0e78ed5eb558\" (UID: \"b3da8c75-360a-438e-9394-0e78ed5eb558\") "
Oct 31 01:57:43.784777 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:43.784665    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3da8c75-360a-438e-9394-0e78ed5eb558-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3da8c75-360a-438e-9394-0e78ed5eb558-tls-certs\") pod \"b3da8c75-360a-438e-9394-0e78ed5eb558\" (UID: \"b3da8c75-360a-438e-9394-0e78ed5eb558\") "
Oct 31 01:57:43.784777 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:43.784710    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-home\" (UniqueName: \"kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-home\") pod \"b3da8c75-360a-438e-9394-0e78ed5eb558\" (UID: \"b3da8c75-360a-438e-9394-0e78ed5eb558\") "
Oct 31 01:57:43.784777 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:43.784749    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-model-cache" pod "b3da8c75-360a-438e-9394-0e78ed5eb558" (UID: "b3da8c75-360a-438e-9394-0e78ed5eb558"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:43.785001 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:43.784868    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-tmp-dir\") pod \"b3da8c75-360a-438e-9394-0e78ed5eb558\" (UID: \"b3da8c75-360a-438e-9394-0e78ed5eb558\") "
Oct 31 01:57:43.785067 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:43.785025    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-home" pod "b3da8c75-360a-438e-9394-0e78ed5eb558" (UID: "b3da8c75-360a-438e-9394-0e78ed5eb558"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:43.785208 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:43.785178    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-tmp-dir" pod "b3da8c75-360a-438e-9394-0e78ed5eb558" (UID: "b3da8c75-360a-438e-9394-0e78ed5eb558"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:43.785860 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:43.785823    5650 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-home\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:43.785860 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:43.785856    5650 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-tmp-dir\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:43.786039 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:43.785872    5650 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-model-cache\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:43.787201 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:43.787175    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3da8c75-360a-438e-9394-0e78ed5eb558-tls-certs" pod "b3da8c75-360a-438e-9394-0e78ed5eb558" (UID: "b3da8c75-360a-438e-9394-0e78ed5eb558"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:57:43.787406 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:43.787378    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3da8c75-360a-438e-9394-0e78ed5eb558-kube-api-access-s8hdh" pod "b3da8c75-360a-438e-9394-0e78ed5eb558" (UID: "b3da8c75-360a-438e-9394-0e78ed5eb558"). InnerVolumeSpecName "kube-api-access-s8hdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:57:43.787516 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:43.787409    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-dshm" pod "b3da8c75-360a-438e-9394-0e78ed5eb558" (UID: "b3da8c75-360a-438e-9394-0e78ed5eb558"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:43.888850 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:43.888806    5650 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3da8c75-360a-438e-9394-0e78ed5eb558-dshm\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:43.888850 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:43.888847    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8hdh\" (UniqueName: \"kubernetes.io/projected/b3da8c75-360a-438e-9394-0e78ed5eb558-kube-api-access-s8hdh\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:43.889109 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:43.888863    5650 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3da8c75-360a-438e-9394-0e78ed5eb558-tls-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:44.058527 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:44.055287    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"]
Oct 31 01:57:44.058527 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:44.055598    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p" podUID="a2c0b4f9-f1b1-45e8-b076-ba5768b15a67" containerName="istio-proxy" containerID="cri-o://cca43b0d2b44ec9c45e4aecfb2d212a6d3c808d5787184610d517980d1e4d06a" gracePeriod=30
Oct 31 01:57:44.454921 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:44.454882    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr" event={"ID":"b3da8c75-360a-438e-9394-0e78ed5eb558","Type":"ContainerDied","Data":"8fbb65d5a777c5ac6b675976c5e7c708fd9255acae0f241ebbb3d455a83755c8"}
Oct 31 01:57:44.455484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:44.455464    5650 scope.go:122] "RemoveContainer" containerID="300dd3cbd99d40f47ea905b9c13af5511599b20fc2a36d856685d119c4a25d0a"
Oct 31 01:57:44.455593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:44.455484    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"
Oct 31 01:57:44.498632 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:44.498597    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"]
Oct 31 01:57:44.509116 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:44.509066    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6d7f5d/auth-test-model-6d7f5d-kserve-6d69c5bc57-4vgnr"]
Oct 31 01:57:45.172669 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:45.172624    5650 patch_prober.go:29] interesting pod/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 31 01:57:45.172841 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:45.172689    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p" podUID="a2c0b4f9-f1b1-45e8-b076-ba5768b15a67" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:57:45.600149 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:45.599719    5650 scope.go:122] "RemoveContainer" containerID="b696cd6e32924d85482e642fdf4d1c31b42d014e9c73c802d4fdaccc9bbfeba5"
Oct 31 01:57:46.121607 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.121574    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3da8c75-360a-438e-9394-0e78ed5eb558" path="/var/lib/kubelet/pods/b3da8c75-360a-438e-9394-0e78ed5eb558/volumes"
Oct 31 01:57:46.490142 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.490102    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m" podUID="0221d5f2-b6f6-4aaf-950a-b1028238ee91" containerName="main" containerID="cri-o://748038ee3c580013c9bcff0841ec21512163838b1af2f0a46521a3fbda880288" gracePeriod=45
Oct 31 01:57:46.792864 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.792836    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:46.873743 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.873702    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn"]
Oct 31 01:57:46.874953 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.874926    5650 cpu_manager.go:414] "containerMap: removing container" podUID="0221d5f2-b6f6-4aaf-950a-b1028238ee91" containerName="main"
Oct 31 01:57:46.874953 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.874954    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0221d5f2-b6f6-4aaf-950a-b1028238ee91" containerName="main"
Oct 31 01:57:46.875166 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.874977    5650 cpu_manager.go:414] "containerMap: removing container" podUID="b3da8c75-360a-438e-9394-0e78ed5eb558" containerName="main"
Oct 31 01:57:46.875166 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.874985    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3da8c75-360a-438e-9394-0e78ed5eb558" containerName="main"
Oct 31 01:57:46.875320 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.875215    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="0221d5f2-b6f6-4aaf-950a-b1028238ee91" containerName="main"
Oct 31 01:57:46.875320 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.875232    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3da8c75-360a-438e-9394-0e78ed5eb558" containerName="main"
Oct 31 01:57:46.875875 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.875850    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0221d5f2-b6f6-4aaf-950a-b1028238ee91-kube-api-access-hhbj2\" (UniqueName: \"kubernetes.io/projected/0221d5f2-b6f6-4aaf-950a-b1028238ee91-kube-api-access-hhbj2\") pod \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\" (UID: \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\") "
Oct 31 01:57:46.876068 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.876049    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-dshm\") pod \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\" (UID: \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\") "
Oct 31 01:57:46.876196 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.876181    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-tmp-dir\") pod \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\" (UID: \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\") "
Oct 31 01:57:46.876264 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.876248    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0221d5f2-b6f6-4aaf-950a-b1028238ee91-tls-certs\" (UniqueName: \"kubernetes.io/secret/0221d5f2-b6f6-4aaf-950a-b1028238ee91-tls-certs\") pod \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\" (UID: \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\") "
Oct 31 01:57:46.876330 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.876313    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-model-cache\") pod \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\" (UID: \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\") "
Oct 31 01:57:46.876429 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.876417    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-home\" (UniqueName: \"kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-home\") pod \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\" (UID: \"0221d5f2-b6f6-4aaf-950a-b1028238ee91\") "
Oct 31 01:57:46.876772 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.876749    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-home" pod "0221d5f2-b6f6-4aaf-950a-b1028238ee91" (UID: "0221d5f2-b6f6-4aaf-950a-b1028238ee91"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:46.876879 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.876765    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-model-cache" pod "0221d5f2-b6f6-4aaf-950a-b1028238ee91" (UID: "0221d5f2-b6f6-4aaf-950a-b1028238ee91"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:46.876962 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.876852    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-tmp-dir" pod "0221d5f2-b6f6-4aaf-950a-b1028238ee91" (UID: "0221d5f2-b6f6-4aaf-950a-b1028238ee91"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:46.877726 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.877688    5650 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-tmp-dir\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:46.877726 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.877718    5650 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-model-cache\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:46.877925 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.877732    5650 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-home\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:46.879150 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.879090    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0221d5f2-b6f6-4aaf-950a-b1028238ee91-tls-certs" pod "0221d5f2-b6f6-4aaf-950a-b1028238ee91" (UID: "0221d5f2-b6f6-4aaf-950a-b1028238ee91"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:57:46.879653 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.879545    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-dshm" pod "0221d5f2-b6f6-4aaf-950a-b1028238ee91" (UID: "0221d5f2-b6f6-4aaf-950a-b1028238ee91"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:46.879896 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.879857    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0221d5f2-b6f6-4aaf-950a-b1028238ee91-kube-api-access-hhbj2" pod "0221d5f2-b6f6-4aaf-950a-b1028238ee91" (UID: "0221d5f2-b6f6-4aaf-950a-b1028238ee91"). InnerVolumeSpecName "kube-api-access-hhbj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:57:46.880089 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.880056    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn"
Oct 31 01:57:46.888352 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.888326    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-f353c2-serving-cert\""
Oct 31 01:57:46.891695 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.891669    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn"]
Oct 31 01:57:46.979290 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.979238    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d826bb9d-d180-4162-b17b-e0a4fc613789-maas-api-tls\") pod \"maas-api-e2e-autores-f353c2-55b8488df6-ljdqn\" (UID: \"d826bb9d-d180-4162-b17b-e0a4fc613789\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn"
Oct 31 01:57:46.979290 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.979291    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d826bb9d-d180-4162-b17b-e0a4fc613789-metrics-certs\") pod \"maas-api-e2e-autores-f353c2-55b8488df6-ljdqn\" (UID: \"d826bb9d-d180-4162-b17b-e0a4fc613789\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn"
Oct 31 01:57:46.979575 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.979318    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt864\" (UniqueName: \"kubernetes.io/projected/d826bb9d-d180-4162-b17b-e0a4fc613789-kube-api-access-lt864\") pod \"maas-api-e2e-autores-f353c2-55b8488df6-ljdqn\" (UID: \"d826bb9d-d180-4162-b17b-e0a4fc613789\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn"
Oct 31 01:57:46.979575 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.979550    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d826bb9d-d180-4162-b17b-e0a4fc613789-openshift-service-ca\") pod \"maas-api-e2e-autores-f353c2-55b8488df6-ljdqn\" (UID: \"d826bb9d-d180-4162-b17b-e0a4fc613789\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn"
Oct 31 01:57:46.979686 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.979673    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhbj2\" (UniqueName: \"kubernetes.io/projected/0221d5f2-b6f6-4aaf-950a-b1028238ee91-kube-api-access-hhbj2\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:46.979728 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.979689    5650 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0221d5f2-b6f6-4aaf-950a-b1028238ee91-dshm\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:46.979728 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:46.979719    5650 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0221d5f2-b6f6-4aaf-950a-b1028238ee91-tls-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:47.080906 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:47.080816    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d826bb9d-d180-4162-b17b-e0a4fc613789-maas-api-tls\") pod \"maas-api-e2e-autores-f353c2-55b8488df6-ljdqn\" (UID: \"d826bb9d-d180-4162-b17b-e0a4fc613789\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn"
Oct 31 01:57:47.080906 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:47.080863    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d826bb9d-d180-4162-b17b-e0a4fc613789-metrics-certs\") pod \"maas-api-e2e-autores-f353c2-55b8488df6-ljdqn\" (UID: \"d826bb9d-d180-4162-b17b-e0a4fc613789\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn"
Oct 31 01:57:47.080906 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:47.080901    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt864\" (UniqueName: \"kubernetes.io/projected/d826bb9d-d180-4162-b17b-e0a4fc613789-kube-api-access-lt864\") pod \"maas-api-e2e-autores-f353c2-55b8488df6-ljdqn\" (UID: \"d826bb9d-d180-4162-b17b-e0a4fc613789\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn"
Oct 31 01:57:47.081256 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:57:47.080987    5650 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-serving-cert: secret "maas-api-e2e-autores-f353c2-serving-cert" not found
Oct 31 01:57:47.081256 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:57:47.081079    5650 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d826bb9d-d180-4162-b17b-e0a4fc613789-maas-api-tls podName:d826bb9d-d180-4162-b17b-e0a4fc613789 nodeName:}" failed. No retries permitted until 2026-10-31 01:57:47.581057771 +0000 UTC m=+2223.584772060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d826bb9d-d180-4162-b17b-e0a4fc613789-maas-api-tls") pod "maas-api-e2e-autores-f353c2-55b8488df6-ljdqn" (UID: "d826bb9d-d180-4162-b17b-e0a4fc613789") : secret "maas-api-e2e-autores-f353c2-serving-cert" not found
Oct 31 01:57:47.081256 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:47.081140    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d826bb9d-d180-4162-b17b-e0a4fc613789-openshift-service-ca\") pod \"maas-api-e2e-autores-f353c2-55b8488df6-ljdqn\" (UID: \"d826bb9d-d180-4162-b17b-e0a4fc613789\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn"
Oct 31 01:57:47.081849 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:47.081816    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d826bb9d-d180-4162-b17b-e0a4fc613789-openshift-service-ca\") pod \"maas-api-e2e-autores-f353c2-55b8488df6-ljdqn\" (UID: \"d826bb9d-d180-4162-b17b-e0a4fc613789\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn"
Oct 31 01:57:47.083814 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:47.083791    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d826bb9d-d180-4162-b17b-e0a4fc613789-metrics-certs\") pod \"maas-api-e2e-autores-f353c2-55b8488df6-ljdqn\" (UID: \"d826bb9d-d180-4162-b17b-e0a4fc613789\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn"
Oct 31 01:57:47.095613 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:47.095584    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt864\" (UniqueName: \"kubernetes.io/projected/d826bb9d-d180-4162-b17b-e0a4fc613789-kube-api-access-lt864\") pod \"maas-api-e2e-autores-f353c2-55b8488df6-ljdqn\" (UID: \"d826bb9d-d180-4162-b17b-e0a4fc613789\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn"
Oct 31 01:57:47.490147 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:47.490114    5650 generic.go:356] "Generic (PLEG): container finished" podID="0221d5f2-b6f6-4aaf-950a-b1028238ee91" containerID="748038ee3c580013c9bcff0841ec21512163838b1af2f0a46521a3fbda880288" exitCode=0
Oct 31 01:57:47.490359 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:47.490166    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"
Oct 31 01:57:47.490359 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:47.490204    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m" event={"ID":"0221d5f2-b6f6-4aaf-950a-b1028238ee91","Type":"ContainerDied","Data":"748038ee3c580013c9bcff0841ec21512163838b1af2f0a46521a3fbda880288"}
Oct 31 01:57:47.490359 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:47.490252    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m" event={"ID":"0221d5f2-b6f6-4aaf-950a-b1028238ee91","Type":"ContainerDied","Data":"9e9d5e3af1028dba035d2de13abbc619aac761c3e645e93cabe17e714ca8549a"}
Oct 31 01:57:47.490359 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:47.490274    5650 scope.go:122] "RemoveContainer" containerID="748038ee3c580013c9bcff0841ec21512163838b1af2f0a46521a3fbda880288"
Oct 31 01:57:47.503693 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:47.503669    5650 scope.go:122] "RemoveContainer" containerID="748038ee3c580013c9bcff0841ec21512163838b1af2f0a46521a3fbda880288"
Oct 31 01:57:47.504025 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:57:47.504003    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"748038ee3c580013c9bcff0841ec21512163838b1af2f0a46521a3fbda880288\": container with ID starting with 748038ee3c580013c9bcff0841ec21512163838b1af2f0a46521a3fbda880288 not found: ID does not exist" containerID="748038ee3c580013c9bcff0841ec21512163838b1af2f0a46521a3fbda880288"
Oct 31 01:57:47.504107 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:47.504032    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"748038ee3c580013c9bcff0841ec21512163838b1af2f0a46521a3fbda880288"} err="failed to get container status \"748038ee3c580013c9bcff0841ec21512163838b1af2f0a46521a3fbda880288\": rpc error: code = NotFound desc = could not find container \"748038ee3c580013c9bcff0841ec21512163838b1af2f0a46521a3fbda880288\": container with ID starting with 748038ee3c580013c9bcff0841ec21512163838b1af2f0a46521a3fbda880288 not found: ID does not exist"
Oct 31 01:57:47.584470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:47.583631    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"]
Oct 31 01:57:47.590473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:47.588219    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-e57add/auth-test-model-e57add-kserve-79d85df557-tpr4m"]
Oct 31 01:57:47.599472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:47.598723    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d826bb9d-d180-4162-b17b-e0a4fc613789-maas-api-tls\") pod \"maas-api-e2e-autores-f353c2-55b8488df6-ljdqn\" (UID: \"d826bb9d-d180-4162-b17b-e0a4fc613789\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn"
Oct 31 01:57:47.642466 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:47.642238    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d826bb9d-d180-4162-b17b-e0a4fc613789-maas-api-tls\") pod \"maas-api-e2e-autores-f353c2-55b8488df6-ljdqn\" (UID: \"d826bb9d-d180-4162-b17b-e0a4fc613789\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn"
Oct 31 01:57:47.843161 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:47.843054    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn"
Oct 31 01:57:48.120425 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:48.120346    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0221d5f2-b6f6-4aaf-950a-b1028238ee91" path="/var/lib/kubelet/pods/0221d5f2-b6f6-4aaf-950a-b1028238ee91/volumes"
Oct 31 01:57:48.136281 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:48.136252    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn"]
Oct 31 01:57:48.136410 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:48.136394    5650 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:57:48.501055 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:48.501019    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn" event={"ID":"d826bb9d-d180-4162-b17b-e0a4fc613789","Type":"ContainerStarted","Data":"22bbec34615aa36b2239b073e41ec43b991b096a7602ed04c4548ad938b7633b"}
Oct 31 01:57:49.246331 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:57:49.246290    5650 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0221d5f2_b6f6_4aaf_950a_b1028238ee91.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c0b4f9_f1b1_45e8_b076_ba5768b15a67.slice/crio-conmon-cca43b0d2b44ec9c45e4aecfb2d212a6d3c808d5787184610d517980d1e4d06a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c0b4f9_f1b1_45e8_b076_ba5768b15a67.slice/crio-cca43b0d2b44ec9c45e4aecfb2d212a6d3c808d5787184610d517980d1e4d06a.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 01:57:49.407501 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.402761    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:49.511582 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.511542    5650 generic.go:356] "Generic (PLEG): container finished" podID="a2c0b4f9-f1b1-45e8-b076-ba5768b15a67" containerID="cca43b0d2b44ec9c45e4aecfb2d212a6d3c808d5787184610d517980d1e4d06a" exitCode=0
Oct 31 01:57:49.511763 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.511592    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"
Oct 31 01:57:49.511763 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.511628    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p" event={"ID":"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67","Type":"ContainerDied","Data":"cca43b0d2b44ec9c45e4aecfb2d212a6d3c808d5787184610d517980d1e4d06a"}
Oct 31 01:57:49.511763 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.511726    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p" event={"ID":"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67","Type":"ContainerDied","Data":"26a2aafc798a2bf35537e18b79f720c1716f2681f3fc1dd1279db952bf7091a1"}
Oct 31 01:57:49.511763 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.511757    5650 scope.go:122] "RemoveContainer" containerID="cca43b0d2b44ec9c45e4aecfb2d212a6d3c808d5787184610d517980d1e4d06a"
Oct 31 01:57:49.513596 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.513570    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn" event={"ID":"d826bb9d-d180-4162-b17b-e0a4fc613789","Type":"ContainerStarted","Data":"09ee73f7d99a202f7d5a2f7e47e222ba698cee2d38ffc45f5c8f0749862dd9c8"}
Oct 31 01:57:49.513712 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.513648    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn"
Oct 31 01:57:49.523027 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.523008    5650 scope.go:122] "RemoveContainer" containerID="cca43b0d2b44ec9c45e4aecfb2d212a6d3c808d5787184610d517980d1e4d06a"
Oct 31 01:57:49.523352 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:57:49.523332    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cca43b0d2b44ec9c45e4aecfb2d212a6d3c808d5787184610d517980d1e4d06a\": container with ID starting with cca43b0d2b44ec9c45e4aecfb2d212a6d3c808d5787184610d517980d1e4d06a not found: ID does not exist" containerID="cca43b0d2b44ec9c45e4aecfb2d212a6d3c808d5787184610d517980d1e4d06a"
Oct 31 01:57:49.523405 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.523362    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cca43b0d2b44ec9c45e4aecfb2d212a6d3c808d5787184610d517980d1e4d06a"} err="failed to get container status \"cca43b0d2b44ec9c45e4aecfb2d212a6d3c808d5787184610d517980d1e4d06a\": rpc error: code = NotFound desc = could not find container \"cca43b0d2b44ec9c45e4aecfb2d212a6d3c808d5787184610d517980d1e4d06a\": container with ID starting with cca43b0d2b44ec9c45e4aecfb2d212a6d3c808d5787184610d517980d1e4d06a not found: ID does not exist"
Oct 31 01:57:49.533017 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.532995    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istiod-ca-cert\") pod \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") "
Oct 31 01:57:49.533145 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.533034    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-workload-certs\") pod \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") "
Oct 31 01:57:49.533145 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.533076    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-workload-socket\") pod \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") "
Oct 31 01:57:49.533283 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.533267    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-podinfo\") pod \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") "
Oct 31 01:57:49.533343 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.533301    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-token\" (UniqueName: \"kubernetes.io/projected/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-token\") pod \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") "
Oct 31 01:57:49.533405 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.533361    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istiod-ca-cert" pod "a2c0b4f9-f1b1-45e8-b076-ba5768b15a67" (UID: "a2c0b4f9-f1b1-45e8-b076-ba5768b15a67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:57:49.533479 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.533400    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-workload-certs" pod "a2c0b4f9-f1b1-45e8-b076-ba5768b15a67" (UID: "a2c0b4f9-f1b1-45e8-b076-ba5768b15a67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:49.533479 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.533431    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-workload-socket" pod "a2c0b4f9-f1b1-45e8-b076-ba5768b15a67" (UID: "a2c0b4f9-f1b1-45e8-b076-ba5768b15a67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:49.533599 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.533492    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-kube-api-access-vjbz5\" (UniqueName: \"kubernetes.io/projected/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-kube-api-access-vjbz5\") pod \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") "
Oct 31 01:57:49.533599 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.533572    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-credential-socket\") pod \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") "
Oct 31 01:57:49.533707 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.533612    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-data\") pod \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") "
Oct 31 01:57:49.533707 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.533645    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-envoy\") pod \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\" (UID: \"a2c0b4f9-f1b1-45e8-b076-ba5768b15a67\") "
Oct 31 01:57:49.533854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.533831    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-credential-socket" pod "a2c0b4f9-f1b1-45e8-b076-ba5768b15a67" (UID: "a2c0b4f9-f1b1-45e8-b076-ba5768b15a67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:49.533976 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.533954    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-data" pod "a2c0b4f9-f1b1-45e8-b076-ba5768b15a67" (UID: "a2c0b4f9-f1b1-45e8-b076-ba5768b15a67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:49.534398 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.534378    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:49.534495 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.534403    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:49.534495 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.534416    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:49.534495 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.534431    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:49.534495 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.534459    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:49.535749 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.535717    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-envoy" pod "a2c0b4f9-f1b1-45e8-b076-ba5768b15a67" (UID: "a2c0b4f9-f1b1-45e8-b076-ba5768b15a67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:49.535839 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.535731    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-kube-api-access-vjbz5" pod "a2c0b4f9-f1b1-45e8-b076-ba5768b15a67" (UID: "a2c0b4f9-f1b1-45e8-b076-ba5768b15a67"). InnerVolumeSpecName "kube-api-access-vjbz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:57:49.535979 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.535956    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-token" pod "a2c0b4f9-f1b1-45e8-b076-ba5768b15a67" (UID: "a2c0b4f9-f1b1-45e8-b076-ba5768b15a67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:57:49.536084 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.536069    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-podinfo" pod "a2c0b4f9-f1b1-45e8-b076-ba5768b15a67" (UID: "a2c0b4f9-f1b1-45e8-b076-ba5768b15a67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:57:49.539314 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.539263    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn" podStartSLOduration=3.109066215 podStartE2EDuration="3.539248918s" totalImagesPullingTime="430.182703ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:57:49.538865257 +0000 UTC m=+2225.542579571" watchObservedRunningTime="2026-10-31 01:57:49.539248918 +0000 UTC m=+2225.542963208"
Oct 31 01:57:49.636393 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.636313    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:49.636393 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.636342    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:49.636393 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.636352    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjbz5\" (UniqueName: \"kubernetes.io/projected/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-kube-api-access-vjbz5\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:49.636393 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.636373    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:57:49.844550 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.844491    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"]
Oct 31 01:57:49.847606 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:49.847579    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-ccfa21-openshift-default-865b6f9d69-8tp2p"]
Oct 31 01:57:50.122175 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:50.122141    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2c0b4f9-f1b1-45e8-b076-ba5768b15a67" path="/var/lib/kubelet/pods/a2c0b4f9-f1b1-45e8-b076-ba5768b15a67/volumes"
Oct 31 01:57:53.534897 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:57:53.534850    5650 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0221d5f2_b6f6_4aaf_950a_b1028238ee91.slice/crio-9e9d5e3af1028dba035d2de13abbc619aac761c3e645e93cabe17e714ca8549a\": RecentStats: unable to find data in memory cache]"
Oct 31 01:57:55.529662 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:57:55.529632    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn"
Oct 31 01:57:58.557656 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:57:58.557616    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:58:01.951289 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:58:01.951246    5650 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0221d5f2_b6f6_4aaf_950a_b1028238ee91.slice/crio-9e9d5e3af1028dba035d2de13abbc619aac761c3e645e93cabe17e714ca8549a\": RecentStats: unable to find data in memory cache]"
Oct 31 01:58:02.646257 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.646192    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"]
Oct 31 01:58:02.651242 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.651206    5650 cpu_manager.go:414] "containerMap: removing container" podUID="a2c0b4f9-f1b1-45e8-b076-ba5768b15a67" containerName="istio-proxy"
Oct 31 01:58:02.651242 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.651242    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2c0b4f9-f1b1-45e8-b076-ba5768b15a67" containerName="istio-proxy"
Oct 31 01:58:02.651473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.651421    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2c0b4f9-f1b1-45e8-b076-ba5768b15a67" containerName="istio-proxy"
Oct 31 01:58:02.674502 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.671798    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"]
Oct 31 01:58:02.674502 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.671911    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:02.677676 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.677650    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-f353c2\"/\"kube-root-ca.crt\""
Oct 31 01:58:02.679092 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.679070    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-f353c2\"/\"default-dockercfg-gwjrj\""
Oct 31 01:58:02.679256 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.679082    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-f353c2\"/\"autores-model-f353c2-kserve-self-signed-certs\""
Oct 31 01:58:02.679374 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.679082    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-f353c2\"/\"openshift-service-ca.crt\""
Oct 31 01:58:02.858469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.848751    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmlgc\" (UniqueName: \"kubernetes.io/projected/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-kube-api-access-qmlgc\") pod \"autores-model-f353c2-kserve-dcd7f8f77-h9ggr\" (UID: \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\") " pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:02.858469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.848833    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-tls-certs\") pod \"autores-model-f353c2-kserve-dcd7f8f77-h9ggr\" (UID: \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\") " pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:02.858469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.848926    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-home\") pod \"autores-model-f353c2-kserve-dcd7f8f77-h9ggr\" (UID: \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\") " pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:02.858469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.848954    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-dshm\") pod \"autores-model-f353c2-kserve-dcd7f8f77-h9ggr\" (UID: \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\") " pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:02.858469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.849072    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-model-cache\") pod \"autores-model-f353c2-kserve-dcd7f8f77-h9ggr\" (UID: \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\") " pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:02.858469 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.849218    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-tmp-dir\") pod \"autores-model-f353c2-kserve-dcd7f8f77-h9ggr\" (UID: \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\") " pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:02.950862 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.950767    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-tls-certs\") pod \"autores-model-f353c2-kserve-dcd7f8f77-h9ggr\" (UID: \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\") " pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:02.951053 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.950866    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-home\") pod \"autores-model-f353c2-kserve-dcd7f8f77-h9ggr\" (UID: \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\") " pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:02.951053 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.950896    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-dshm\") pod \"autores-model-f353c2-kserve-dcd7f8f77-h9ggr\" (UID: \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\") " pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:02.951053 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.951007    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-model-cache\") pod \"autores-model-f353c2-kserve-dcd7f8f77-h9ggr\" (UID: \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\") " pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:02.951253 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.951096    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-tmp-dir\") pod \"autores-model-f353c2-kserve-dcd7f8f77-h9ggr\" (UID: \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\") " pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:02.951253 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.951213    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmlgc\" (UniqueName: \"kubernetes.io/projected/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-kube-api-access-qmlgc\") pod \"autores-model-f353c2-kserve-dcd7f8f77-h9ggr\" (UID: \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\") " pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:02.951848 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.951820    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-model-cache\") pod \"autores-model-f353c2-kserve-dcd7f8f77-h9ggr\" (UID: \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\") " pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:02.952262 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.951870    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-tmp-dir\") pod \"autores-model-f353c2-kserve-dcd7f8f77-h9ggr\" (UID: \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\") " pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:02.952262 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.951990    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-home\") pod \"autores-model-f353c2-kserve-dcd7f8f77-h9ggr\" (UID: \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\") " pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:02.954360 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.954332    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-tls-certs\") pod \"autores-model-f353c2-kserve-dcd7f8f77-h9ggr\" (UID: \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\") " pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:02.954488 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.954342    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-dshm\") pod \"autores-model-f353c2-kserve-dcd7f8f77-h9ggr\" (UID: \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\") " pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:02.964555 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.964523    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmlgc\" (UniqueName: \"kubernetes.io/projected/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-kube-api-access-qmlgc\") pod \"autores-model-f353c2-kserve-dcd7f8f77-h9ggr\" (UID: \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\") " pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:02.987504 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:02.987422    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:03.296244 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:58:03.296204    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d5e8d84_0e67_4809_bbd9_8f3f55ce58c9.slice/crio-de794f9c9501a30c2524942f9382f58396f898df51c0ea0a6e52438e6e2b18af WatchSource:0}: Error finding container de794f9c9501a30c2524942f9382f58396f898df51c0ea0a6e52438e6e2b18af: Status 404 returned error can't find the container with id de794f9c9501a30c2524942f9382f58396f898df51c0ea0a6e52438e6e2b18af
Oct 31 01:58:03.297489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:03.297416    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"]
Oct 31 01:58:03.555208 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:58:03.555124    5650 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0221d5f2_b6f6_4aaf_950a_b1028238ee91.slice/crio-9e9d5e3af1028dba035d2de13abbc619aac761c3e645e93cabe17e714ca8549a\": RecentStats: unable to find data in memory cache]"
Oct 31 01:58:03.681605 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:03.681571    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr" event={"ID":"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9","Type":"ContainerStarted","Data":"c2c9380b82fbdc1561a59f313c76f74da6ed4e0a7ef91e874e8d5675d1689a9c"}
Oct 31 01:58:03.681819 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:03.681614    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr" event={"ID":"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9","Type":"ContainerStarted","Data":"de794f9c9501a30c2524942f9382f58396f898df51c0ea0a6e52438e6e2b18af"}
Oct 31 01:58:03.707116 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:03.707064    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr" podStartSLOduration=1.7070507830000001 podStartE2EDuration="1.707050783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:58:03.70284566 +0000 UTC m=+2239.706559952" watchObservedRunningTime="2026-10-31 01:58:03.707050783 +0000 UTC m=+2239.710765073"
Oct 31 01:58:03.988293 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:03.988254    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:12.992466 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:12.989476    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:13.008727 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:13.008697    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:13.754720 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:58:13.754678    5650 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0221d5f2_b6f6_4aaf_950a_b1028238ee91.slice/crio-9e9d5e3af1028dba035d2de13abbc619aac761c3e645e93cabe17e714ca8549a\": RecentStats: unable to find data in memory cache]"
Oct 31 01:58:13.798251 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:13.798219    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:16.837594 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:58:16.837538    5650 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0221d5f2_b6f6_4aaf_950a_b1028238ee91.slice/crio-9e9d5e3af1028dba035d2de13abbc619aac761c3e645e93cabe17e714ca8549a\": RecentStats: unable to find data in memory cache]"
Oct 31 01:58:19.280339 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:19.280295    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"]
Oct 31 01:58:23.854233 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:58:23.854193    5650 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0221d5f2_b6f6_4aaf_950a_b1028238ee91.slice/crio-9e9d5e3af1028dba035d2de13abbc619aac761c3e645e93cabe17e714ca8549a\": RecentStats: unable to find data in memory cache]"
Oct 31 01:58:24.026991 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:24.026949    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f488f08-kr8ds"]
Oct 31 01:58:24.032520 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:24.032487    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f488f08-kr8ds"]
Oct 31 01:58:24.121859 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:24.121760    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16b1ef53-8ed9-4180-9086-e60ce2c50dc7" path="/var/lib/kubelet/pods/16b1ef53-8ed9-4180-9086-e60ce2c50dc7/volumes"
Oct 31 01:58:28.557415 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:58:28.557323    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:58:30.535065 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.535024    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn"]
Oct 31 01:58:30.535579 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.535228    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn" podUID="d826bb9d-d180-4162-b17b-e0a4fc613789" containerName="maas-api" containerID="cri-o://09ee73f7d99a202f7d5a2f7e47e222ba698cee2d38ffc45f5c8f0749862dd9c8" gracePeriod=30
Oct 31 01:58:30.737164 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.737126    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"]
Oct 31 01:58:30.742342 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.742217    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:30.754523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.753385    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"]
Oct 31 01:58:30.754523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.753395    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-5555cb-openshift-default-dockercfg-vctpm\""
Oct 31 01:58:30.843340 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.843306    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-podinfo\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:30.843556 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.843360    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvb76\" (UniqueName: \"kubernetes.io/projected/17e6d59f-a66f-4c5c-aff8-8725deabb93f-kube-api-access-hvb76\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:30.843622 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.843566    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-workload-socket\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:30.843622 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.843614    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-data\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:30.843794 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.843720    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-credential-socket\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:30.843794 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.843760    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istiod-ca-cert\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:30.843900 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.843822    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-token\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:30.843955 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.843911    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-workload-certs\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:30.843955 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.843928    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-envoy\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:30.929483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.929381    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn"
Oct 31 01:58:30.972467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.948535    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-workload-socket\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:30.972467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.948626    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-data\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:30.972467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.948664    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-credential-socket\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:30.972467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.948692    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istiod-ca-cert\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:30.972467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.948748    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-token\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:30.972467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.948837    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-workload-certs\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:30.972467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.948864    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-envoy\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:30.972467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.948996    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-podinfo\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:30.972467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.949045    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvb76\" (UniqueName: \"kubernetes.io/projected/17e6d59f-a66f-4c5c-aff8-8725deabb93f-kube-api-access-hvb76\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:30.972467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.949978    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-workload-socket\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:30.972467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.950250    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-workload-certs\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:30.972467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.966691    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-credential-socket\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:30.972467 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:30.966977    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-data\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:31.011251 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.009907    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvb76\" (UniqueName: \"kubernetes.io/projected/17e6d59f-a66f-4c5c-aff8-8725deabb93f-kube-api-access-hvb76\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:31.022490 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.014940    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istiod-ca-cert\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:31.026555 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.025518    5650 generic.go:356] "Generic (PLEG): container finished" podID="d826bb9d-d180-4162-b17b-e0a4fc613789" containerID="09ee73f7d99a202f7d5a2f7e47e222ba698cee2d38ffc45f5c8f0749862dd9c8" exitCode=0
Oct 31 01:58:31.026555 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.025652    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn" event={"ID":"d826bb9d-d180-4162-b17b-e0a4fc613789","Type":"ContainerDied","Data":"09ee73f7d99a202f7d5a2f7e47e222ba698cee2d38ffc45f5c8f0749862dd9c8"}
Oct 31 01:58:31.026555 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.025705    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn" event={"ID":"d826bb9d-d180-4162-b17b-e0a4fc613789","Type":"ContainerDied","Data":"22bbec34615aa36b2239b073e41ec43b991b096a7602ed04c4548ad938b7633b"}
Oct 31 01:58:31.026555 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.025737    5650 scope.go:122] "RemoveContainer" containerID="09ee73f7d99a202f7d5a2f7e47e222ba698cee2d38ffc45f5c8f0749862dd9c8"
Oct 31 01:58:31.026555 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.025908    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn"
Oct 31 01:58:31.032503 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.029711    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-podinfo\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:31.046526 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.045296    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-token\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:31.084483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.081081    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d826bb9d-d180-4162-b17b-e0a4fc613789-kube-api-access-lt864\" (UniqueName: \"kubernetes.io/projected/d826bb9d-d180-4162-b17b-e0a4fc613789-kube-api-access-lt864\") pod \"d826bb9d-d180-4162-b17b-e0a4fc613789\" (UID: \"d826bb9d-d180-4162-b17b-e0a4fc613789\") "
Oct 31 01:58:31.084483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.081198    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d826bb9d-d180-4162-b17b-e0a4fc613789-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d826bb9d-d180-4162-b17b-e0a4fc613789-metrics-certs\") pod \"d826bb9d-d180-4162-b17b-e0a4fc613789\" (UID: \"d826bb9d-d180-4162-b17b-e0a4fc613789\") "
Oct 31 01:58:31.084483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.081277    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d826bb9d-d180-4162-b17b-e0a4fc613789-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d826bb9d-d180-4162-b17b-e0a4fc613789-openshift-service-ca\") pod \"d826bb9d-d180-4162-b17b-e0a4fc613789\" (UID: \"d826bb9d-d180-4162-b17b-e0a4fc613789\") "
Oct 31 01:58:31.084483 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.081596    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d826bb9d-d180-4162-b17b-e0a4fc613789-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d826bb9d-d180-4162-b17b-e0a4fc613789-maas-api-tls\") pod \"d826bb9d-d180-4162-b17b-e0a4fc613789\" (UID: \"d826bb9d-d180-4162-b17b-e0a4fc613789\") "
Oct 31 01:58:31.088524 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.088402    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d826bb9d-d180-4162-b17b-e0a4fc613789-openshift-service-ca" pod "d826bb9d-d180-4162-b17b-e0a4fc613789" (UID: "d826bb9d-d180-4162-b17b-e0a4fc613789"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:58:31.186052 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.185849    5650 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d826bb9d-d180-4162-b17b-e0a4fc613789-openshift-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:31.186052 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.184414    5650 scope.go:122] "RemoveContainer" containerID="09ee73f7d99a202f7d5a2f7e47e222ba698cee2d38ffc45f5c8f0749862dd9c8"
Oct 31 01:58:31.190475 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.187486    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d826bb9d-d180-4162-b17b-e0a4fc613789-metrics-certs" pod "d826bb9d-d180-4162-b17b-e0a4fc613789" (UID: "d826bb9d-d180-4162-b17b-e0a4fc613789"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:58:31.190475 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.187630    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d826bb9d-d180-4162-b17b-e0a4fc613789-kube-api-access-lt864" pod "d826bb9d-d180-4162-b17b-e0a4fc613789" (UID: "d826bb9d-d180-4162-b17b-e0a4fc613789"). InnerVolumeSpecName "kube-api-access-lt864". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:58:31.190475 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.187646    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d826bb9d-d180-4162-b17b-e0a4fc613789-maas-api-tls" pod "d826bb9d-d180-4162-b17b-e0a4fc613789" (UID: "d826bb9d-d180-4162-b17b-e0a4fc613789"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:58:31.190475 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.187842    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-envoy\") pod \"e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") " pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:31.193790 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:58:31.193358    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09ee73f7d99a202f7d5a2f7e47e222ba698cee2d38ffc45f5c8f0749862dd9c8\": container with ID starting with 09ee73f7d99a202f7d5a2f7e47e222ba698cee2d38ffc45f5c8f0749862dd9c8 not found: ID does not exist" containerID="09ee73f7d99a202f7d5a2f7e47e222ba698cee2d38ffc45f5c8f0749862dd9c8"
Oct 31 01:58:31.193790 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.193462    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09ee73f7d99a202f7d5a2f7e47e222ba698cee2d38ffc45f5c8f0749862dd9c8"} err="failed to get container status \"09ee73f7d99a202f7d5a2f7e47e222ba698cee2d38ffc45f5c8f0749862dd9c8\": rpc error: code = NotFound desc = could not find container \"09ee73f7d99a202f7d5a2f7e47e222ba698cee2d38ffc45f5c8f0749862dd9c8\": container with ID starting with 09ee73f7d99a202f7d5a2f7e47e222ba698cee2d38ffc45f5c8f0749862dd9c8 not found: ID does not exist"
Oct 31 01:58:31.287946 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.287900    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d826bb9d-d180-4162-b17b-e0a4fc613789-maas-api-tls\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:31.287946 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.287940    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt864\" (UniqueName: \"kubernetes.io/projected/d826bb9d-d180-4162-b17b-e0a4fc613789-kube-api-access-lt864\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:31.287946 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.287952    5650 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d826bb9d-d180-4162-b17b-e0a4fc613789-metrics-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:31.356153 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.356046    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn"]
Oct 31 01:58:31.361941 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.361906    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f353c2-55b8488df6-ljdqn"]
Oct 31 01:58:31.365954 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.365918    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:31.686180 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:58:31.686146    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e6d59f_a66f_4c5c_aff8_8725deabb93f.slice/crio-a29b026738a98da55583be257a70e42514d3102b6ddf322ef1f4ab9a23fa2fa2 WatchSource:0}: Error finding container a29b026738a98da55583be257a70e42514d3102b6ddf322ef1f4ab9a23fa2fa2: Status 404 returned error can't find the container with id a29b026738a98da55583be257a70e42514d3102b6ddf322ef1f4ab9a23fa2fa2
Oct 31 01:58:31.687102 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.687080    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"]
Oct 31 01:58:31.689085 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.689054    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:58:31.689171 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.689129    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:58:31.689171 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:31.689159    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:58:31.982362 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:58:31.982327    5650 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0221d5f2_b6f6_4aaf_950a_b1028238ee91.slice/crio-9e9d5e3af1028dba035d2de13abbc619aac761c3e645e93cabe17e714ca8549a\": RecentStats: unable to find data in memory cache]"
Oct 31 01:58:32.037998 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:32.037960    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m" event={"ID":"17e6d59f-a66f-4c5c-aff8-8725deabb93f","Type":"ContainerStarted","Data":"d01fa1c2f3c95eb99577cfdf4e8a180ad2c60310fd15f68d170fc4b0e15383c2"}
Oct 31 01:58:32.038184 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:32.038018    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m" event={"ID":"17e6d59f-a66f-4c5c-aff8-8725deabb93f","Type":"ContainerStarted","Data":"a29b026738a98da55583be257a70e42514d3102b6ddf322ef1f4ab9a23fa2fa2"}
Oct 31 01:58:32.067428 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:32.067363    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m" podStartSLOduration=2.067344917 podStartE2EDuration="2.067344917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:58:32.062841989 +0000 UTC m=+2268.066556282" watchObservedRunningTime="2026-10-31 01:58:32.067344917 +0000 UTC m=+2268.071059208"
Oct 31 01:58:32.120432 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:32.120398    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d826bb9d-d180-4162-b17b-e0a4fc613789" path="/var/lib/kubelet/pods/d826bb9d-d180-4162-b17b-e0a4fc613789/volumes"
Oct 31 01:58:32.367137 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:32.367050    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:32.373340 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:32.373312    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:33.046936 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:33.046895    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:33.048157 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:33.048136    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:58:33.645576 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:58:33.645532    5650 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0221d5f2_b6f6_4aaf_950a_b1028238ee91.slice/crio-9e9d5e3af1028dba035d2de13abbc619aac761c3e645e93cabe17e714ca8549a\": RecentStats: unable to find data in memory cache]"
Oct 31 01:58:33.645788 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:58:33.645688    5650 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0221d5f2_b6f6_4aaf_950a_b1028238ee91.slice/crio-9e9d5e3af1028dba035d2de13abbc619aac761c3e645e93cabe17e714ca8549a\": RecentStats: unable to find data in memory cache]"
Oct 31 01:58:33.872308 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:58:33.872264    5650 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0221d5f2_b6f6_4aaf_950a_b1028238ee91.slice/crio-9e9d5e3af1028dba035d2de13abbc619aac761c3e645e93cabe17e714ca8549a\": RecentStats: unable to find data in memory cache]"
Oct 31 01:58:34.315657 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:34.315610    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr" podUID="7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9" containerName="main" containerID="cri-o://c2c9380b82fbdc1561a59f313c76f74da6ed4e0a7ef91e874e8d5675d1689a9c" gracePeriod=45
Oct 31 01:58:34.619186 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:34.619153    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:34.770523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:34.764798    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-dshm\") pod \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\" (UID: \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\") "
Oct 31 01:58:34.770523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:34.764905    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-kube-api-access-qmlgc\" (UniqueName: \"kubernetes.io/projected/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-kube-api-access-qmlgc\") pod \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\" (UID: \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\") "
Oct 31 01:58:34.770523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:34.764984    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-tls-certs\") pod \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\" (UID: \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\") "
Oct 31 01:58:34.770523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:34.765038    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-model-cache\") pod \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\" (UID: \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\") "
Oct 31 01:58:34.770523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:34.765117    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-home\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-home\") pod \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\" (UID: \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\") "
Oct 31 01:58:34.770523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:34.767734    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-tmp-dir\") pod \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\" (UID: \"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9\") "
Oct 31 01:58:34.770523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:34.769015    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-tmp-dir" pod "7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9" (UID: "7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:58:34.770523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:34.769264    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-home" pod "7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9" (UID: "7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:58:34.770523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:34.769380    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-model-cache" pod "7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9" (UID: "7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:58:34.773332 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:34.772150    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-dshm" pod "7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9" (UID: "7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:58:34.775823 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:34.775629    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-tls-certs" pod "7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9" (UID: "7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:58:34.775823 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:34.775633    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-kube-api-access-qmlgc" pod "7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9" (UID: "7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9"). InnerVolumeSpecName "kube-api-access-qmlgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:58:34.870755 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:34.870651    5650 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-tmp-dir\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:34.870755 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:34.870682    5650 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-dshm\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:34.870755 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:34.870695    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmlgc\" (UniqueName: \"kubernetes.io/projected/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-kube-api-access-qmlgc\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:34.870755 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:34.870710    5650 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-tls-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:34.870755 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:34.870724    5650 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-model-cache\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:34.870755 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:34.870738    5650 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9-home\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 01:58:35.066774 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.066737    5650 generic.go:356] "Generic (PLEG): container finished" podID="7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9" containerID="c2c9380b82fbdc1561a59f313c76f74da6ed4e0a7ef91e874e8d5675d1689a9c" exitCode=0
Oct 31 01:58:35.066963 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.066784    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"
Oct 31 01:58:35.066963 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.066810    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr" event={"ID":"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9","Type":"ContainerDied","Data":"c2c9380b82fbdc1561a59f313c76f74da6ed4e0a7ef91e874e8d5675d1689a9c"}
Oct 31 01:58:35.066963 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.066858    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr" event={"ID":"7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9","Type":"ContainerDied","Data":"de794f9c9501a30c2524942f9382f58396f898df51c0ea0a6e52438e6e2b18af"}
Oct 31 01:58:35.066963 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.066876    5650 scope.go:122] "RemoveContainer" containerID="c2c9380b82fbdc1561a59f313c76f74da6ed4e0a7ef91e874e8d5675d1689a9c"
Oct 31 01:58:35.078959 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.078938    5650 scope.go:122] "RemoveContainer" containerID="c2c9380b82fbdc1561a59f313c76f74da6ed4e0a7ef91e874e8d5675d1689a9c"
Oct 31 01:58:35.079258 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:58:35.079233    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c9380b82fbdc1561a59f313c76f74da6ed4e0a7ef91e874e8d5675d1689a9c\": container with ID starting with c2c9380b82fbdc1561a59f313c76f74da6ed4e0a7ef91e874e8d5675d1689a9c not found: ID does not exist" containerID="c2c9380b82fbdc1561a59f313c76f74da6ed4e0a7ef91e874e8d5675d1689a9c"
Oct 31 01:58:35.079350 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.079266    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c9380b82fbdc1561a59f313c76f74da6ed4e0a7ef91e874e8d5675d1689a9c"} err="failed to get container status \"c2c9380b82fbdc1561a59f313c76f74da6ed4e0a7ef91e874e8d5675d1689a9c\": rpc error: code = NotFound desc = could not find container \"c2c9380b82fbdc1561a59f313c76f74da6ed4e0a7ef91e874e8d5675d1689a9c\": container with ID starting with c2c9380b82fbdc1561a59f313c76f74da6ed4e0a7ef91e874e8d5675d1689a9c not found: ID does not exist"
Oct 31 01:58:35.098881 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.098843    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"]
Oct 31 01:58:35.105344 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.105302    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-f353c2/autores-model-f353c2-kserve-dcd7f8f77-h9ggr"]
Oct 31 01:58:35.822417 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.822373    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw"]
Oct 31 01:58:35.823489 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.823463    5650 cpu_manager.go:414] "containerMap: removing container" podUID="d826bb9d-d180-4162-b17b-e0a4fc613789" containerName="maas-api"
Oct 31 01:58:35.823593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.823489    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d826bb9d-d180-4162-b17b-e0a4fc613789" containerName="maas-api"
Oct 31 01:58:35.823593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.823575    5650 cpu_manager.go:414] "containerMap: removing container" podUID="7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9" containerName="main"
Oct 31 01:58:35.823593 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.823585    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9" containerName="main"
Oct 31 01:58:35.823791 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.823776    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9" containerName="main"
Oct 31 01:58:35.823847 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.823802    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="d826bb9d-d180-4162-b17b-e0a4fc613789" containerName="maas-api"
Oct 31 01:58:35.829332 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.829309    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw"
Oct 31 01:58:35.860783 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.859865    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-5555cb\""
Oct 31 01:58:35.860783 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.859973    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-5555cb-dockercfg-jmp98\""
Oct 31 01:58:35.866113 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.866083    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct"]
Oct 31 01:58:35.870883 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.870832    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct"
Oct 31 01:58:35.877626 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.877601    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw"]
Oct 31 01:58:35.906601 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.906571    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct"]
Oct 31 01:58:35.996472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.990901    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hlxt\" (UniqueName: \"kubernetes.io/projected/218ba464-377e-48e6-a904-59aac3818f28-kube-api-access-5hlxt\") pod \"payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct\" (UID: \"218ba464-377e-48e6-a904-59aac3818f28\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct"
Oct 31 01:58:35.996472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.990979    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/218ba464-377e-48e6-a904-59aac3818f28-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct\" (UID: \"218ba464-377e-48e6-a904-59aac3818f28\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct"
Oct 31 01:58:35.996472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.991073    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6lf6\" (UniqueName: \"kubernetes.io/projected/9fc48192-d1e7-4f3d-8285-3c0d98465241-kube-api-access-n6lf6\") pod \"payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw\" (UID: \"9fc48192-d1e7-4f3d-8285-3c0d98465241\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw"
Oct 31 01:58:35.996472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:35.991185    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc48192-d1e7-4f3d-8285-3c0d98465241-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw\" (UID: \"9fc48192-d1e7-4f3d-8285-3c0d98465241\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw"
Oct 31 01:58:36.176519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:36.172659    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc48192-d1e7-4f3d-8285-3c0d98465241-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw\" (UID: \"9fc48192-d1e7-4f3d-8285-3c0d98465241\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw"
Oct 31 01:58:36.176519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:36.173056    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hlxt\" (UniqueName: \"kubernetes.io/projected/218ba464-377e-48e6-a904-59aac3818f28-kube-api-access-5hlxt\") pod \"payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct\" (UID: \"218ba464-377e-48e6-a904-59aac3818f28\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct"
Oct 31 01:58:36.176519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:36.173114    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/218ba464-377e-48e6-a904-59aac3818f28-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct\" (UID: \"218ba464-377e-48e6-a904-59aac3818f28\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct"
Oct 31 01:58:36.176519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:36.173229    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6lf6\" (UniqueName: \"kubernetes.io/projected/9fc48192-d1e7-4f3d-8285-3c0d98465241-kube-api-access-n6lf6\") pod \"payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw\" (UID: \"9fc48192-d1e7-4f3d-8285-3c0d98465241\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw"
Oct 31 01:58:36.176519 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:36.174594    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc48192-d1e7-4f3d-8285-3c0d98465241-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw\" (UID: \"9fc48192-d1e7-4f3d-8285-3c0d98465241\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw"
Oct 31 01:58:36.178086 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:36.178057    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/218ba464-377e-48e6-a904-59aac3818f28-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct\" (UID: \"218ba464-377e-48e6-a904-59aac3818f28\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct"
Oct 31 01:58:36.235419 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:36.235384    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6lf6\" (UniqueName: \"kubernetes.io/projected/9fc48192-d1e7-4f3d-8285-3c0d98465241-kube-api-access-n6lf6\") pod \"payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw\" (UID: \"9fc48192-d1e7-4f3d-8285-3c0d98465241\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw"
Oct 31 01:58:36.247702 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:36.246775    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hlxt\" (UniqueName: \"kubernetes.io/projected/218ba464-377e-48e6-a904-59aac3818f28-kube-api-access-5hlxt\") pod \"payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct\" (UID: \"218ba464-377e-48e6-a904-59aac3818f28\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct"
Oct 31 01:58:36.252124 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:36.251998    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9" path="/var/lib/kubelet/pods/7d5e8d84-0e67-4809-bbd9-8f3f55ce58c9/volumes"
Oct 31 01:58:36.445947 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:36.445858    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw"
Oct 31 01:58:36.491661 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:36.491625    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct"
Oct 31 01:58:36.761198 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:36.759959    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw"]
Oct 31 01:58:36.763274 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:58:36.763241    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc48192_d1e7_4f3d_8285_3c0d98465241.slice/crio-31af3d7e075a92e009494d80afc3d51280d11db925af2b76081abec5bfaf4f21 WatchSource:0}: Error finding container 31af3d7e075a92e009494d80afc3d51280d11db925af2b76081abec5bfaf4f21: Status 404 returned error can't find the container with id 31af3d7e075a92e009494d80afc3d51280d11db925af2b76081abec5bfaf4f21
Oct 31 01:58:36.920176 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:58:36.920136    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218ba464_377e_48e6_a904_59aac3818f28.slice/crio-5327dab662093f9321241b3e80641e9be56f9d92ba473d0ff19bf7818026b17c WatchSource:0}: Error finding container 5327dab662093f9321241b3e80641e9be56f9d92ba473d0ff19bf7818026b17c: Status 404 returned error can't find the container with id 5327dab662093f9321241b3e80641e9be56f9d92ba473d0ff19bf7818026b17c
Oct 31 01:58:36.922051 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:36.922016    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct"]
Oct 31 01:58:37.359208 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:37.358969    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct" event={"ID":"218ba464-377e-48e6-a904-59aac3818f28","Type":"ContainerStarted","Data":"863072f7e6633458c783512a207f7c89ec36a773dea14b3e288a763c06d44930"}
Oct 31 01:58:37.359208 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:37.359080    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct" event={"ID":"218ba464-377e-48e6-a904-59aac3818f28","Type":"ContainerStarted","Data":"5327dab662093f9321241b3e80641e9be56f9d92ba473d0ff19bf7818026b17c"}
Oct 31 01:58:37.361859 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:37.361792    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct"
Oct 31 01:58:37.365579 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:37.364118    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw" event={"ID":"9fc48192-d1e7-4f3d-8285-3c0d98465241","Type":"ContainerStarted","Data":"e9e36b549647e6d265e3691453d27f7bf1e8f76db88665263719225772fb9e96"}
Oct 31 01:58:37.365579 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:37.364175    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw" event={"ID":"9fc48192-d1e7-4f3d-8285-3c0d98465241","Type":"ContainerStarted","Data":"31af3d7e075a92e009494d80afc3d51280d11db925af2b76081abec5bfaf4f21"}
Oct 31 01:58:37.365579 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:37.365132    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw"
Oct 31 01:58:37.394844 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:37.393814    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct" podStartSLOduration=2.393797601 podStartE2EDuration="2.393797601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:58:37.384848307 +0000 UTC m=+2273.388562600" watchObservedRunningTime="2026-10-31 01:58:37.393797601 +0000 UTC m=+2273.397511892"
Oct 31 01:58:37.413043 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:37.412988    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw" podStartSLOduration=2.412970819 podStartE2EDuration="2.412970819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:58:37.408774688 +0000 UTC m=+2273.412488980" watchObservedRunningTime="2026-10-31 01:58:37.412970819 +0000 UTC m=+2273.416685147"
Oct 31 01:58:44.016373 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:58:44.016331    5650 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0221d5f2_b6f6_4aaf_950a_b1028238ee91.slice/crio-9e9d5e3af1028dba035d2de13abbc619aac761c3e645e93cabe17e714ca8549a\": RecentStats: unable to find data in memory cache]"
Oct 31 01:58:45.671350 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:45.671121    5650 scope.go:122] "RemoveContainer" containerID="483d7abc0bb3d588e2fa0530e76682ab6027a3a530ca8ddcb07bf175ef9c7d09"
Oct 31 01:58:48.028613 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:48.028576    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6407d3cc-dpppj"]
Oct 31 01:58:48.032462 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:48.032417    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6407d3cc-dpppj"]
Oct 31 01:58:48.120543 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:48.120511    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21539936-858a-45d4-8dcf-c4e15813af0c" path="/var/lib/kubelet/pods/21539936-858a-45d4-8dcf-c4e15813af0c/volumes"
Oct 31 01:58:48.376733 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:48.376653    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct"
Oct 31 01:58:49.414803 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:49.414766    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw"
Oct 31 01:58:56.749545 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:56.749501    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb"]
Oct 31 01:58:56.755681 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:56.755642    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb"
Oct 31 01:58:56.762482 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:56.762382    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv"]
Oct 31 01:58:56.768899 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:56.768862    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-ac7e0d-dockercfg-wm8j4\""
Oct 31 01:58:56.770579 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:56.770547    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-ac7e0d\""
Oct 31 01:58:56.798268 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:56.798232    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv"
Oct 31 01:58:56.799202 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:56.799073    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb"]
Oct 31 01:58:56.799202 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:56.799141    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv"]
Oct 31 01:58:56.893470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:56.892922    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/652c2106-a19b-4498-9ed2-c3775bcca730-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb\" (UID: \"652c2106-a19b-4498-9ed2-c3775bcca730\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb"
Oct 31 01:58:56.893470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:56.893049    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdhz9\" (UniqueName: \"kubernetes.io/projected/652c2106-a19b-4498-9ed2-c3775bcca730-kube-api-access-wdhz9\") pod \"payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb\" (UID: \"652c2106-a19b-4498-9ed2-c3775bcca730\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb"
Oct 31 01:58:57.014352 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:57.011554    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdhz9\" (UniqueName: \"kubernetes.io/projected/652c2106-a19b-4498-9ed2-c3775bcca730-kube-api-access-wdhz9\") pod \"payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb\" (UID: \"652c2106-a19b-4498-9ed2-c3775bcca730\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb"
Oct 31 01:58:57.014352 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:57.011828    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxnxq\" (UniqueName: \"kubernetes.io/projected/5d87a817-5974-457b-b870-45ce974b5148-kube-api-access-dxnxq\") pod \"payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv\" (UID: \"5d87a817-5974-457b-b870-45ce974b5148\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv"
Oct 31 01:58:57.014352 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:57.012004    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/652c2106-a19b-4498-9ed2-c3775bcca730-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb\" (UID: \"652c2106-a19b-4498-9ed2-c3775bcca730\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb"
Oct 31 01:58:57.014352 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:57.012056    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d87a817-5974-457b-b870-45ce974b5148-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv\" (UID: \"5d87a817-5974-457b-b870-45ce974b5148\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv"
Oct 31 01:58:57.014352 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:57.013684    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/652c2106-a19b-4498-9ed2-c3775bcca730-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb\" (UID: \"652c2106-a19b-4498-9ed2-c3775bcca730\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb"
Oct 31 01:58:57.080484 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:57.080427    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdhz9\" (UniqueName: \"kubernetes.io/projected/652c2106-a19b-4498-9ed2-c3775bcca730-kube-api-access-wdhz9\") pod \"payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb\" (UID: \"652c2106-a19b-4498-9ed2-c3775bcca730\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb"
Oct 31 01:58:57.149470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:57.116527    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d87a817-5974-457b-b870-45ce974b5148-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv\" (UID: \"5d87a817-5974-457b-b870-45ce974b5148\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv"
Oct 31 01:58:57.177752 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:57.175867    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxnxq\" (UniqueName: \"kubernetes.io/projected/5d87a817-5974-457b-b870-45ce974b5148-kube-api-access-dxnxq\") pod \"payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv\" (UID: \"5d87a817-5974-457b-b870-45ce974b5148\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv"
Oct 31 01:58:57.177752 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:57.168865    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb"
Oct 31 01:58:57.227480 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:57.219312    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d87a817-5974-457b-b870-45ce974b5148-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv\" (UID: \"5d87a817-5974-457b-b870-45ce974b5148\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv"
Oct 31 01:58:57.253795 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:57.248189    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxnxq\" (UniqueName: \"kubernetes.io/projected/5d87a817-5974-457b-b870-45ce974b5148-kube-api-access-dxnxq\") pod \"payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv\" (UID: \"5d87a817-5974-457b-b870-45ce974b5148\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv"
Oct 31 01:58:57.430023 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:57.428760    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv"
Oct 31 01:58:57.552104 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:58:57.552058    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652c2106_a19b_4498_9ed2_c3775bcca730.slice/crio-dd8c7cc0b87ad58902c4b50c969ccb27ff242d134718c3f86071d13b2c61ac55 WatchSource:0}: Error finding container dd8c7cc0b87ad58902c4b50c969ccb27ff242d134718c3f86071d13b2c61ac55: Status 404 returned error can't find the container with id dd8c7cc0b87ad58902c4b50c969ccb27ff242d134718c3f86071d13b2c61ac55
Oct 31 01:58:57.576797 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:57.576739    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb"]
Oct 31 01:58:57.739854 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:57.739826    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv"]
Oct 31 01:58:57.809161 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:57.809124    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv" event={"ID":"5d87a817-5974-457b-b870-45ce974b5148","Type":"ContainerStarted","Data":"3f3ed209b5bf4e502fc8eab5f1abb2f453f8af098af1bffb17f28f7da41b4ba2"}
Oct 31 01:58:57.811280 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:57.811197    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb" event={"ID":"652c2106-a19b-4498-9ed2-c3775bcca730","Type":"ContainerStarted","Data":"001ad07148689e3e3c9d2e2eb4070ae14fe958561126a3a68cff692cf57775ec"}
Oct 31 01:58:57.811280 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:57.811250    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb" event={"ID":"652c2106-a19b-4498-9ed2-c3775bcca730","Type":"ContainerStarted","Data":"dd8c7cc0b87ad58902c4b50c969ccb27ff242d134718c3f86071d13b2c61ac55"}
Oct 31 01:58:57.811500 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:57.811283    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb"
Oct 31 01:58:57.834132 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:57.834062    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb" podStartSLOduration=1.83404072 podStartE2EDuration="1.83404072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:58:57.830384184 +0000 UTC m=+2293.834098476" watchObservedRunningTime="2026-10-31 01:58:57.83404072 +0000 UTC m=+2293.837755017"
Oct 31 01:58:58.555752 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:58:58.555710    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:58:58.821858 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:58.821750    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv" event={"ID":"5d87a817-5974-457b-b870-45ce974b5148","Type":"ContainerStarted","Data":"950f0614ddc7369253b40b7313eaa18a36741a47ed6708b34501089589a97f20"}
Oct 31 01:58:58.821858 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:58.821839    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv"
Oct 31 01:58:58.869470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:58:58.863714    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv" podStartSLOduration=2.863689907 podStartE2EDuration="2.863689907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:58:58.862936133 +0000 UTC m=+2294.866650426" watchObservedRunningTime="2026-10-31 01:58:58.863689907 +0000 UTC m=+2294.867404199"
Oct 31 01:59:00.566906 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.566871    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"]
Oct 31 01:59:00.572646 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.572619    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.582744 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.582702    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-89e955-openshift-default-dockercfg-7cq7z\""
Oct 31 01:59:00.585147 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.585116    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"]
Oct 31 01:59:00.640169 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.640133    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-workload-socket\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.640360 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.640197    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-token\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.640360 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.640272    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-envoy\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.640486 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.640427    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-podinfo\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.640537 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.640527    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5d4w\" (UniqueName: \"kubernetes.io/projected/4238ae64-8bce-4c8d-8f14-339706f25bed-kube-api-access-s5d4w\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.640603 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.640588    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-credential-socket\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.640666 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.640651    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-workload-certs\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.640777 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.640757    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4238ae64-8bce-4c8d-8f14-339706f25bed-istiod-ca-cert\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.640860 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.640818    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-data\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.742655 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.742122    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-data\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.742937 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.742913    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-workload-socket\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.743421 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.743400    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-token\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.754216 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.754183    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-envoy\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.762302 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.743309    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-workload-socket\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.762759 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.742609    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-data\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.762759 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.762379    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-podinfo\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.763044 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.763020    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5d4w\" (UniqueName: \"kubernetes.io/projected/4238ae64-8bce-4c8d-8f14-339706f25bed-kube-api-access-s5d4w\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.763259 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.763239    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-credential-socket\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.763412 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.763396    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-workload-certs\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.763668 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.763646    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4238ae64-8bce-4c8d-8f14-339706f25bed-istiod-ca-cert\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.764666 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.764609    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4238ae64-8bce-4c8d-8f14-339706f25bed-istiod-ca-cert\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.765703 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.765357    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-workload-certs\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.765846 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.765631    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-credential-socket\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.797777 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.797691    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-token\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.816661 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.816629    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-podinfo\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.817292 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.817206    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-envoy\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.834473 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.827556    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5d4w\" (UniqueName: \"kubernetes.io/projected/4238ae64-8bce-4c8d-8f14-339706f25bed-kube-api-access-s5d4w\") pod \"e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") " pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:00.913485 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:00.902302    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:01.234300 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:59:01.233832    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4238ae64_8bce_4c8d_8f14_339706f25bed.slice/crio-ba2af375dda50fcec9c245e668e5afeeb2ae41fd3174182e2fc16689e48230ce WatchSource:0}: Error finding container ba2af375dda50fcec9c245e668e5afeeb2ae41fd3174182e2fc16689e48230ce: Status 404 returned error can't find the container with id ba2af375dda50fcec9c245e668e5afeeb2ae41fd3174182e2fc16689e48230ce
Oct 31 01:59:01.236391 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:01.235682    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"]
Oct 31 01:59:01.241989 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:01.238127    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:59:01.241989 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:01.238229    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:59:01.241989 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:01.238273    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:59:01.914522 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:01.914476    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn" event={"ID":"4238ae64-8bce-4c8d-8f14-339706f25bed","Type":"ContainerStarted","Data":"63b85f5e86c8fd464c93877f608e1e6afb85c30df583e60a3906d48b361ad2c1"}
Oct 31 01:59:01.915123 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:01.914553    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn" event={"ID":"4238ae64-8bce-4c8d-8f14-339706f25bed","Type":"ContainerStarted","Data":"ba2af375dda50fcec9c245e668e5afeeb2ae41fd3174182e2fc16689e48230ce"}
Oct 31 01:59:01.943776 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:01.943022    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn" podStartSLOduration=1.9430022500000002 podStartE2EDuration="1.94300225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:59:01.941157905 +0000 UTC m=+2297.944872197" watchObservedRunningTime="2026-10-31 01:59:01.94300225 +0000 UTC m=+2297.946716546"
Oct 31 01:59:02.903758 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:02.903718    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:02.910238 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:02.910207    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:02.922824 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:02.922787    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:02.923808 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:02.923788    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:05.625986 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:05.625943    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6"]
Oct 31 01:59:05.638759 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:05.638726    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6"
Oct 31 01:59:05.639986 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:05.639951    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6"]
Oct 31 01:59:05.747468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:05.736970    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-89e955-dockercfg-9pm2k\""
Oct 31 01:59:05.747468 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:05.737311    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-89e955\""
Oct 31 01:59:05.798543 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:05.797614    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816cf08b-0669-44c0-a959-53c3be786f1d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6\" (UID: \"816cf08b-0669-44c0-a959-53c3be786f1d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6"
Oct 31 01:59:05.798543 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:05.797951    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btddh\" (UniqueName: \"kubernetes.io/projected/816cf08b-0669-44c0-a959-53c3be786f1d-kube-api-access-btddh\") pod \"payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6\" (UID: \"816cf08b-0669-44c0-a959-53c3be786f1d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6"
Oct 31 01:59:05.902528 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:05.901721    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btddh\" (UniqueName: \"kubernetes.io/projected/816cf08b-0669-44c0-a959-53c3be786f1d-kube-api-access-btddh\") pod \"payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6\" (UID: \"816cf08b-0669-44c0-a959-53c3be786f1d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6"
Oct 31 01:59:05.902528 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:05.901885    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816cf08b-0669-44c0-a959-53c3be786f1d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6\" (UID: \"816cf08b-0669-44c0-a959-53c3be786f1d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6"
Oct 31 01:59:05.903096 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:05.903064    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816cf08b-0669-44c0-a959-53c3be786f1d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6\" (UID: \"816cf08b-0669-44c0-a959-53c3be786f1d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6"
Oct 31 01:59:05.926534 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:05.926496    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btddh\" (UniqueName: \"kubernetes.io/projected/816cf08b-0669-44c0-a959-53c3be786f1d-kube-api-access-btddh\") pod \"payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6\" (UID: \"816cf08b-0669-44c0-a959-53c3be786f1d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6"
Oct 31 01:59:05.969274 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:05.967868    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6"
Oct 31 01:59:06.283203 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:06.274254    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6"]
Oct 31 01:59:07.000103 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:06.997035    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6" event={"ID":"816cf08b-0669-44c0-a959-53c3be786f1d","Type":"ContainerStarted","Data":"d9c735d0d8a595b2eaec311e6e16f6a9a48ed23fab362245552e9a01114b399a"}
Oct 31 01:59:07.000103 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:06.997092    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6" event={"ID":"816cf08b-0669-44c0-a959-53c3be786f1d","Type":"ContainerStarted","Data":"f8da060d83675f4da044efb456d445fc836371287436f851ad8d8b357a3deeaa"}
Oct 31 01:59:07.000103 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:06.998349    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6"
Oct 31 01:59:07.027847 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:07.027768    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6" podStartSLOduration=2.027748405 podStartE2EDuration="2.027748405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:59:07.02046904 +0000 UTC m=+2303.024183334" watchObservedRunningTime="2026-10-31 01:59:07.027748405 +0000 UTC m=+2303.031462708"
Oct 31 01:59:08.825816 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:08.825781    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb"
Oct 31 01:59:09.835117 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:09.835086    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv"
Oct 31 01:59:19.016686 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:19.016649    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6"
Oct 31 01:59:19.089640 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:19.089600    5650 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:59:19.089804 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:19.089659    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m" podUID="17e6d59f-a66f-4c5c-aff8-8725deabb93f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:59:23.144057 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:23.143971    5650 generic.go:356] "Generic (PLEG): container finished" podID="17e6d59f-a66f-4c5c-aff8-8725deabb93f" containerID="d01fa1c2f3c95eb99577cfdf4e8a180ad2c60310fd15f68d170fc4b0e15383c2" exitCode=0
Oct 31 01:59:23.144057 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:23.144039    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m" event={"ID":"17e6d59f-a66f-4c5c-aff8-8725deabb93f","Type":"ContainerDied","Data":"d01fa1c2f3c95eb99577cfdf4e8a180ad2c60310fd15f68d170fc4b0e15383c2"}
Oct 31 01:59:23.144596 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:23.144578    5650 scope.go:122] "RemoveContainer" containerID="d01fa1c2f3c95eb99577cfdf4e8a180ad2c60310fd15f68d170fc4b0e15383c2"
Oct 31 01:59:23.146537 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:23.146510    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:59:23.146634 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:23.146582    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:59:23.146634 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:23.146615    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:59:24.211470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:24.204705    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m" event={"ID":"17e6d59f-a66f-4c5c-aff8-8725deabb93f","Type":"ContainerStarted","Data":"d9882cb0bd8d18f1b22605da5a609e49f80bfdf426cbb280e8f9c1369234a920"}
Oct 31 01:59:24.366126 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:24.366081    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:59:24.366348 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:24.366311    5650 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 31 01:59:24.366441 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:24.366362    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m" podUID="17e6d59f-a66f-4c5c-aff8-8725deabb93f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 31 01:59:25.366630 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:25.366596    5650 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 31 01:59:25.367016 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:25.366649    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m" podUID="17e6d59f-a66f-4c5c-aff8-8725deabb93f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 31 01:59:26.367182 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:26.367140    5650 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 31 01:59:26.367614 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:26.367200    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m" podUID="17e6d59f-a66f-4c5c-aff8-8725deabb93f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 31 01:59:27.366870 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:27.366829    5650 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 31 01:59:27.367060 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:27.366890    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m" podUID="17e6d59f-a66f-4c5c-aff8-8725deabb93f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 31 01:59:27.729934 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:27.729890    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2"]
Oct 31 01:59:27.733503 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:27.733474    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2"
Oct 31 01:59:27.738690 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:27.738659    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-0b6abf\""
Oct 31 01:59:27.740363 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:27.740336    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-0b6abf-dockercfg-lhxmm\""
Oct 31 01:59:27.743594 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:27.743565    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2"]
Oct 31 01:59:27.786050 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:27.786008    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0840a2-aff1-4bb6-bba8-968c0ee0e277-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2\" (UID: \"6f0840a2-aff1-4bb6-bba8-968c0ee0e277\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2"
Oct 31 01:59:27.786260 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:27.786147    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxczw\" (UniqueName: \"kubernetes.io/projected/6f0840a2-aff1-4bb6-bba8-968c0ee0e277-kube-api-access-lxczw\") pod \"payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2\" (UID: \"6f0840a2-aff1-4bb6-bba8-968c0ee0e277\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2"
Oct 31 01:59:27.929471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:27.904588    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxczw\" (UniqueName: \"kubernetes.io/projected/6f0840a2-aff1-4bb6-bba8-968c0ee0e277-kube-api-access-lxczw\") pod \"payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2\" (UID: \"6f0840a2-aff1-4bb6-bba8-968c0ee0e277\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2"
Oct 31 01:59:27.990091 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:27.989986    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0840a2-aff1-4bb6-bba8-968c0ee0e277-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2\" (UID: \"6f0840a2-aff1-4bb6-bba8-968c0ee0e277\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2"
Oct 31 01:59:28.028032 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:27.991899    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0840a2-aff1-4bb6-bba8-968c0ee0e277-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2\" (UID: \"6f0840a2-aff1-4bb6-bba8-968c0ee0e277\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2"
Oct 31 01:59:28.191196 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:28.190829    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxczw\" (UniqueName: \"kubernetes.io/projected/6f0840a2-aff1-4bb6-bba8-968c0ee0e277-kube-api-access-lxczw\") pod \"payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2\" (UID: \"6f0840a2-aff1-4bb6-bba8-968c0ee0e277\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2"
Oct 31 01:59:28.358244 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:28.357234    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2"
Oct 31 01:59:28.368040 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:28.368006    5650 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 31 01:59:28.368793 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:28.368297    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m" podUID="17e6d59f-a66f-4c5c-aff8-8725deabb93f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 31 01:59:28.596494 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:59:28.593614    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:59:28.731607 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:59:28.731576    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0840a2_aff1_4bb6_bba8_968c0ee0e277.slice/crio-52998d680ded63864282ff95ca3442cc62c38667304a5f82b7de0e695eb98ea3 WatchSource:0}: Error finding container 52998d680ded63864282ff95ca3442cc62c38667304a5f82b7de0e695eb98ea3: Status 404 returned error can't find the container with id 52998d680ded63864282ff95ca3442cc62c38667304a5f82b7de0e695eb98ea3
Oct 31 01:59:28.732363 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:28.732338    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2"]
Oct 31 01:59:29.293284 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:29.293238    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2" event={"ID":"6f0840a2-aff1-4bb6-bba8-968c0ee0e277","Type":"ContainerStarted","Data":"01ac5716fee527fe7ee4d5017bf280005ff763c34f1969cbc2098feed6624dfd"}
Oct 31 01:59:29.293486 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:29.293312    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2" event={"ID":"6f0840a2-aff1-4bb6-bba8-968c0ee0e277","Type":"ContainerStarted","Data":"52998d680ded63864282ff95ca3442cc62c38667304a5f82b7de0e695eb98ea3"}
Oct 31 01:59:29.293681 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:29.293662    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2"
Oct 31 01:59:29.345470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:29.344854    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2" podStartSLOduration=2.344834964 podStartE2EDuration="2.344834964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:59:29.343986869 +0000 UTC m=+2325.347701161" watchObservedRunningTime="2026-10-31 01:59:29.344834964 +0000 UTC m=+2325.348549257"
Oct 31 01:59:29.367080 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:29.367042    5650 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 31 01:59:29.367266 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:29.367100    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m" podUID="17e6d59f-a66f-4c5c-aff8-8725deabb93f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 31 01:59:30.371763 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:30.371728    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:59:30.372252 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:30.372158    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:59:30.372969 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:30.372951    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 01:59:32.527010 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.526972    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"]
Oct 31 01:59:32.532572 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.532535    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.539968 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.539936    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-20538a-openshift-default-dockercfg-jzr22\""
Oct 31 01:59:32.545150 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.545107    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"]
Oct 31 01:59:32.643814 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.643769    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-workload-certs\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.643814 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.643815    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-workload-socket\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.644045 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.643924    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-istio-envoy\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.644045 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.643951    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxktp\" (UniqueName: \"kubernetes.io/projected/1da683f3-ae35-4c60-a119-7711ca882f54-kube-api-access-qxktp\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.644125 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.644104    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-istio-data\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.644168 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.644158    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1da683f3-ae35-4c60-a119-7711ca882f54-istio-podinfo\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.644233 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.644215    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1da683f3-ae35-4c60-a119-7711ca882f54-istio-token\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.644349 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.644335    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-credential-socket\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.644441 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.644430    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1da683f3-ae35-4c60-a119-7711ca882f54-istiod-ca-cert\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.760470 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.749305    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-istio-data\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.761287 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.761262    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1da683f3-ae35-4c60-a119-7711ca882f54-istio-podinfo\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.761591 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.761474    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1da683f3-ae35-4c60-a119-7711ca882f54-istio-token\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.761760 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.761741    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-credential-socket\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.761901 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.761883    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1da683f3-ae35-4c60-a119-7711ca882f54-istiod-ca-cert\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.762083 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.762034    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-workload-certs\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.762152 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.762109    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-workload-socket\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.762213 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.762172    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-istio-envoy\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.762213 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.762201    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxktp\" (UniqueName: \"kubernetes.io/projected/1da683f3-ae35-4c60-a119-7711ca882f54-kube-api-access-qxktp\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.762333 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.761144    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-istio-data\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.767472 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.764304    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-workload-certs\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.768405 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.768375    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1da683f3-ae35-4c60-a119-7711ca882f54-istiod-ca-cert\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.772898 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.772863    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-credential-socket\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.779858 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.773347    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-workload-socket\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.804505 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.794092    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1da683f3-ae35-4c60-a119-7711ca882f54-istio-token\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.804802 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.794288    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxktp\" (UniqueName: \"kubernetes.io/projected/1da683f3-ae35-4c60-a119-7711ca882f54-kube-api-access-qxktp\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.807745 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.807705    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1da683f3-ae35-4c60-a119-7711ca882f54-istio-podinfo\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.812653 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.812582    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-istio-envoy\") pod \"e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") " pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:32.858194 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:32.857855    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:33.262163 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:33.261985    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"]
Oct 31 01:59:33.266215 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:59:33.266185    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da683f3_ae35_4c60_a119_7711ca882f54.slice/crio-0e3bcac15db3a3626861a2fca287f88da15f08f514be710ef24118c87ab2d84e WatchSource:0}: Error finding container 0e3bcac15db3a3626861a2fca287f88da15f08f514be710ef24118c87ab2d84e: Status 404 returned error can't find the container with id 0e3bcac15db3a3626861a2fca287f88da15f08f514be710ef24118c87ab2d84e
Oct 31 01:59:33.269223 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:33.269185    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:59:33.269346 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:33.269275    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:59:33.269412 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:33.269341    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:59:33.330182 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:33.330145    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww" event={"ID":"1da683f3-ae35-4c60-a119-7711ca882f54","Type":"ContainerStarted","Data":"0e3bcac15db3a3626861a2fca287f88da15f08f514be710ef24118c87ab2d84e"}
Oct 31 01:59:34.339692 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:34.339652    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww" event={"ID":"1da683f3-ae35-4c60-a119-7711ca882f54","Type":"ContainerStarted","Data":"06dcb568530b690aec267324abd0782c765b29d05cd80cab6a092952e8877d97"}
Oct 31 01:59:34.404851 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:34.404787    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww" podStartSLOduration=2.404769093 podStartE2EDuration="2.404769093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:59:34.372630909 +0000 UTC m=+2330.376345201" watchObservedRunningTime="2026-10-31 01:59:34.404769093 +0000 UTC m=+2330.408483384"
Oct 31 01:59:34.861364 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:34.861327    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:34.867523 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:34.867495    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:35.348209 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:35.348176    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:35.349236 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:35.349210    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 01:59:37.109538 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:37.088318    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bae1d5ce-8mx2n"]
Oct 31 01:59:37.147620 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:37.147583    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bae1d5ce-8mx2n"]
Oct 31 01:59:38.074319 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:38.074279    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl"]
Oct 31 01:59:38.080037 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:38.080003    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl"
Oct 31 01:59:38.087180 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:38.087141    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-20538a\""
Oct 31 01:59:38.087180 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:38.087141    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-20538a-dockercfg-mp5k9\""
Oct 31 01:59:38.088580 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:38.088546    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl"]
Oct 31 01:59:38.132012 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:38.131970    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="758dae21-5a65-4610-a7e9-9fd29f6021a9" path="/var/lib/kubelet/pods/758dae21-5a65-4610-a7e9-9fd29f6021a9/volumes"
Oct 31 01:59:38.132553 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:38.132531    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk"]
Oct 31 01:59:38.138024 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:38.138003    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk"]
Oct 31 01:59:38.138158 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:38.138101    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk"
Oct 31 01:59:38.187312 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:38.187261    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx685\" (UniqueName: \"kubernetes.io/projected/82a717a2-e5de-46ea-9bba-6d28baa9a9ad-kube-api-access-hx685\") pod \"payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl\" (UID: \"82a717a2-e5de-46ea-9bba-6d28baa9a9ad\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl"
Oct 31 01:59:38.187733 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:38.187704    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7262357-a521-49d3-88f4-86e8392931b9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk\" (UID: \"f7262357-a521-49d3-88f4-86e8392931b9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk"
Oct 31 01:59:38.188161 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:38.188130    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7lr6\" (UniqueName: \"kubernetes.io/projected/f7262357-a521-49d3-88f4-86e8392931b9-kube-api-access-w7lr6\") pod \"payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk\" (UID: \"f7262357-a521-49d3-88f4-86e8392931b9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk"
Oct 31 01:59:38.188435 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:38.188415    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82a717a2-e5de-46ea-9bba-6d28baa9a9ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl\" (UID: \"82a717a2-e5de-46ea-9bba-6d28baa9a9ad\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl"
Oct 31 01:59:38.325533 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:38.315604    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx685\" (UniqueName: \"kubernetes.io/projected/82a717a2-e5de-46ea-9bba-6d28baa9a9ad-kube-api-access-hx685\") pod \"payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl\" (UID: \"82a717a2-e5de-46ea-9bba-6d28baa9a9ad\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl"
Oct 31 01:59:38.325940 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:38.325905    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7262357-a521-49d3-88f4-86e8392931b9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk\" (UID: \"f7262357-a521-49d3-88f4-86e8392931b9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk"
Oct 31 01:59:38.369471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:38.326364    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7lr6\" (UniqueName: \"kubernetes.io/projected/f7262357-a521-49d3-88f4-86e8392931b9-kube-api-access-w7lr6\") pod \"payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk\" (UID: \"f7262357-a521-49d3-88f4-86e8392931b9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk"
Oct 31 01:59:38.369890 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:38.369853    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82a717a2-e5de-46ea-9bba-6d28baa9a9ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl\" (UID: \"82a717a2-e5de-46ea-9bba-6d28baa9a9ad\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl"
Oct 31 01:59:38.382471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:38.372348    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82a717a2-e5de-46ea-9bba-6d28baa9a9ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl\" (UID: \"82a717a2-e5de-46ea-9bba-6d28baa9a9ad\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl"
Oct 31 01:59:38.441689 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:38.383661    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7262357-a521-49d3-88f4-86e8392931b9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk\" (UID: \"f7262357-a521-49d3-88f4-86e8392931b9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk"
Oct 31 01:59:38.447679 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:38.447642    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7lr6\" (UniqueName: \"kubernetes.io/projected/f7262357-a521-49d3-88f4-86e8392931b9-kube-api-access-w7lr6\") pod \"payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk\" (UID: \"f7262357-a521-49d3-88f4-86e8392931b9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk"
Oct 31 01:59:38.447929 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:38.447679    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx685\" (UniqueName: \"kubernetes.io/projected/82a717a2-e5de-46ea-9bba-6d28baa9a9ad-kube-api-access-hx685\") pod \"payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl\" (UID: \"82a717a2-e5de-46ea-9bba-6d28baa9a9ad\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl"
Oct 31 01:59:38.484239 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:38.484203    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk"
Oct 31 01:59:38.705871 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:38.705836    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl"
Oct 31 01:59:38.792823 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:59:38.792770    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7262357_a521_49d3_88f4_86e8392931b9.slice/crio-653853d5250e02030a7722a9c589a56277460039e5a64ad9037c9153770d4ef1 WatchSource:0}: Error finding container 653853d5250e02030a7722a9c589a56277460039e5a64ad9037c9153770d4ef1: Status 404 returned error can't find the container with id 653853d5250e02030a7722a9c589a56277460039e5a64ad9037c9153770d4ef1
Oct 31 01:59:38.793507 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:38.793474    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk"]
Oct 31 01:59:39.001966 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:59:39.001923    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a717a2_e5de_46ea_9bba_6d28baa9a9ad.slice/crio-9cffc2ed0d3a197eb5b9cd21a42429e3020048f354f7b98d3267f13e936fe8e5 WatchSource:0}: Error finding container 9cffc2ed0d3a197eb5b9cd21a42429e3020048f354f7b98d3267f13e936fe8e5: Status 404 returned error can't find the container with id 9cffc2ed0d3a197eb5b9cd21a42429e3020048f354f7b98d3267f13e936fe8e5
Oct 31 01:59:39.002141 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:39.001974    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl"]
Oct 31 01:59:39.445866 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:39.445759    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl" event={"ID":"82a717a2-e5de-46ea-9bba-6d28baa9a9ad","Type":"ContainerStarted","Data":"80ca25fc89e0abcdf1bf0031fa9457ade4ec14206a36464dbd89e7b3c7b1618e"}
Oct 31 01:59:39.445866 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:39.445828    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl" event={"ID":"82a717a2-e5de-46ea-9bba-6d28baa9a9ad","Type":"ContainerStarted","Data":"9cffc2ed0d3a197eb5b9cd21a42429e3020048f354f7b98d3267f13e936fe8e5"}
Oct 31 01:59:39.445866 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:39.445858    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl"
Oct 31 01:59:39.448072 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:39.448041    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk" event={"ID":"f7262357-a521-49d3-88f4-86e8392931b9","Type":"ContainerStarted","Data":"2c221ac93e1c1f775ac1ecf2b3e962eb96969448554e703133080278cb649d4e"}
Oct 31 01:59:39.448198 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:39.448095    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk" event={"ID":"f7262357-a521-49d3-88f4-86e8392931b9","Type":"ContainerStarted","Data":"653853d5250e02030a7722a9c589a56277460039e5a64ad9037c9153770d4ef1"}
Oct 31 01:59:39.448198 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:39.448120    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk"
Oct 31 01:59:39.470031 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:39.469978    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl" podStartSLOduration=1.4699626239999999 podStartE2EDuration="1.469962624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:59:39.469145073 +0000 UTC m=+2335.472859361" watchObservedRunningTime="2026-10-31 01:59:39.469962624 +0000 UTC m=+2335.473676914"
Oct 31 01:59:39.496283 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:39.496223    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk" podStartSLOduration=1.496206782 podStartE2EDuration="1.496206782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:59:39.491195598 +0000 UTC m=+2335.494909903" watchObservedRunningTime="2026-10-31 01:59:39.496206782 +0000 UTC m=+2335.499921073"
Oct 31 01:59:40.306083 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:40.306050    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2"
Oct 31 01:59:45.789306 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:45.789261    5650 scope.go:122] "RemoveContainer" containerID="11707442b47e158cb31d6558bd08a3de40796b4ddceff9c404d7be1f57bb321e"
Oct 31 01:59:45.847588 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:45.847541    5650 scope.go:122] "RemoveContainer" containerID="7e4383e7a682ffd7a5c97308c248bcd606b927a4084d6cf37d14c614d56c9d99"
Oct 31 01:59:49.057935 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:49.057896    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw"]
Oct 31 01:59:49.066037 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:49.065999    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw"
Oct 31 01:59:49.072177 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:49.071331    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-20538a-serving-cert\""
Oct 31 01:59:49.077028 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:49.076806    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw"]
Oct 31 01:59:49.252170 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:49.251848    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd57354a-e4f0-42fc-a537-336d346ffee0-metrics-certs\") pod \"maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw\" (UID: \"dd57354a-e4f0-42fc-a537-336d346ffee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw"
Oct 31 01:59:49.252170 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:49.252026    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nczwv\" (UniqueName: \"kubernetes.io/projected/dd57354a-e4f0-42fc-a537-336d346ffee0-kube-api-access-nczwv\") pod \"maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw\" (UID: \"dd57354a-e4f0-42fc-a537-336d346ffee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw"
Oct 31 01:59:49.257836 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:49.257794    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd57354a-e4f0-42fc-a537-336d346ffee0-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw\" (UID: \"dd57354a-e4f0-42fc-a537-336d346ffee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw"
Oct 31 01:59:49.258162 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:49.258139    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd57354a-e4f0-42fc-a537-336d346ffee0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw\" (UID: \"dd57354a-e4f0-42fc-a537-336d346ffee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw"
Oct 31 01:59:49.360012 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:49.359906    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd57354a-e4f0-42fc-a537-336d346ffee0-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw\" (UID: \"dd57354a-e4f0-42fc-a537-336d346ffee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw"
Oct 31 01:59:49.360012 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:49.359971    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd57354a-e4f0-42fc-a537-336d346ffee0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw\" (UID: \"dd57354a-e4f0-42fc-a537-336d346ffee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw"
Oct 31 01:59:49.360264 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:49.360033    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd57354a-e4f0-42fc-a537-336d346ffee0-metrics-certs\") pod \"maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw\" (UID: \"dd57354a-e4f0-42fc-a537-336d346ffee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw"
Oct 31 01:59:49.360264 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:49.360125    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nczwv\" (UniqueName: \"kubernetes.io/projected/dd57354a-e4f0-42fc-a537-336d346ffee0-kube-api-access-nczwv\") pod \"maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw\" (UID: \"dd57354a-e4f0-42fc-a537-336d346ffee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw"
Oct 31 01:59:49.360761 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:49.360733    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd57354a-e4f0-42fc-a537-336d346ffee0-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw\" (UID: \"dd57354a-e4f0-42fc-a537-336d346ffee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw"
Oct 31 01:59:49.362609 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:49.362586    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd57354a-e4f0-42fc-a537-336d346ffee0-metrics-certs\") pod \"maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw\" (UID: \"dd57354a-e4f0-42fc-a537-336d346ffee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw"
Oct 31 01:59:49.362746 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:49.362720    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd57354a-e4f0-42fc-a537-336d346ffee0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw\" (UID: \"dd57354a-e4f0-42fc-a537-336d346ffee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw"
Oct 31 01:59:49.374253 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:49.374224    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nczwv\" (UniqueName: \"kubernetes.io/projected/dd57354a-e4f0-42fc-a537-336d346ffee0-kube-api-access-nczwv\") pod \"maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw\" (UID: \"dd57354a-e4f0-42fc-a537-336d346ffee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw"
Oct 31 01:59:49.388107 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:49.387803    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw"
Oct 31 01:59:49.964912 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:49.964876    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw"]
Oct 31 01:59:49.965075 ip-10-0-133-98 kubenswrapper[5650]: W1031 01:59:49.964883    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd57354a_e4f0_42fc_a537_336d346ffee0.slice/crio-ea899212c01da9ab04f4a84fd02ed66659764874822a038d848209a3dd5499d2 WatchSource:0}: Error finding container ea899212c01da9ab04f4a84fd02ed66659764874822a038d848209a3dd5499d2: Status 404 returned error can't find the container with id ea899212c01da9ab04f4a84fd02ed66659764874822a038d848209a3dd5499d2
Oct 31 01:59:50.466415 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:50.466381    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl"
Oct 31 01:59:50.468071 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:50.468048    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk"
Oct 31 01:59:50.810327 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:50.810011    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw" event={"ID":"dd57354a-e4f0-42fc-a537-336d346ffee0","Type":"ContainerStarted","Data":"0d309825e2abc9cd62b6a566bef818ade7733278d160f0f9fe7b79695dfa77d9"}
Oct 31 01:59:50.810602 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:50.810365    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw"
Oct 31 01:59:50.810602 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:50.810390    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw" event={"ID":"dd57354a-e4f0-42fc-a537-336d346ffee0","Type":"ContainerStarted","Data":"ea899212c01da9ab04f4a84fd02ed66659764874822a038d848209a3dd5499d2"}
Oct 31 01:59:50.844753 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:50.844604    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw" podStartSLOduration=1.373475279 podStartE2EDuration="1.844580988s" totalImagesPullingTime="471.105709ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:59:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:59:50.837506622 +0000 UTC m=+2346.841220915" watchObservedRunningTime="2026-10-31 01:59:50.844580988 +0000 UTC m=+2346.848295280"
Oct 31 01:59:55.906033 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:55.905885    5650 generic.go:356] "Generic (PLEG): container finished" podID="4238ae64-8bce-4c8d-8f14-339706f25bed" containerID="63b85f5e86c8fd464c93877f608e1e6afb85c30df583e60a3906d48b361ad2c1" exitCode=0
Oct 31 01:59:55.906784 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:55.906355    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn" event={"ID":"4238ae64-8bce-4c8d-8f14-339706f25bed","Type":"ContainerDied","Data":"63b85f5e86c8fd464c93877f608e1e6afb85c30df583e60a3906d48b361ad2c1"}
Oct 31 01:59:55.907072 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:55.907049    5650 scope.go:122] "RemoveContainer" containerID="63b85f5e86c8fd464c93877f608e1e6afb85c30df583e60a3906d48b361ad2c1"
Oct 31 01:59:55.909946 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:55.909900    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:59:55.910086 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:55.910007    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:59:55.910086 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:55.910064    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 01:59:56.825972 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:56.825939    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw"
Oct 31 01:59:56.920888 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:56.920840    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn" event={"ID":"4238ae64-8bce-4c8d-8f14-339706f25bed","Type":"ContainerStarted","Data":"fbd7c9b84063b4d6a7134c102f907a50a2d84e9e76efc2f830d83daad3c36736"}
Oct 31 01:59:57.925499 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:57.922959    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 01:59:57.926044 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:57.925639    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 31 01:59:57.926044 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:57.925720    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn" podUID="4238ae64-8bce-4c8d-8f14-339706f25bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 31 01:59:58.563079 ip-10-0-133-98 kubenswrapper[5650]: E1031 01:59:58.563029    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 01:59:58.908530 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:58.906142    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 31 01:59:58.908530 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:58.906205    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn" podUID="4238ae64-8bce-4c8d-8f14-339706f25bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 31 01:59:59.909471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:59.908872    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 31 01:59:59.909471 ip-10-0-133-98 kubenswrapper[5650]: I1031 01:59:59.908933    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn" podUID="4238ae64-8bce-4c8d-8f14-339706f25bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 31 02:00:00.383437 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.383386    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw"]
Oct 31 02:00:00.388879 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.388848    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw"]
Oct 31 02:00:00.389076 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.388895    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg"]
Oct 31 02:00:00.389076 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.388984    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw"
Oct 31 02:00:00.393555 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.393501    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp"]
Oct 31 02:00:00.393767 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.393587    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg"
Oct 31 02:00:00.394665 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.394640    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:00:00.397926 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.397899    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp"
Oct 31 02:00:00.426643 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.426599    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2"]
Oct 31 02:00:00.444926 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.444890    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg"]
Oct 31 02:00:00.445207 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.445187    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6"]
Oct 31 02:00:00.450316 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.450285    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6"
Oct 31 02:00:00.451040 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.451016    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2"
Oct 31 02:00:00.471441 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.471409    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp"]
Oct 31 02:00:00.484191 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.484104    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2"]
Oct 31 02:00:00.488713 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.488678    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw"]
Oct 31 02:00:00.498584 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.495157    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw"
Oct 31 02:00:00.498584 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.496231    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh"]
Oct 31 02:00:00.510545 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.506495    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf"]
Oct 31 02:00:00.514564 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.514496    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh"
Oct 31 02:00:00.538394 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.538354    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf"
Oct 31 02:00:00.538581 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.538249    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh"]
Oct 31 02:00:00.538633 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.538584    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw"]
Oct 31 02:00:00.538633 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.538615    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49"]
Oct 31 02:00:00.545764 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.543642    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf"]
Oct 31 02:00:00.545764 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.543689    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49"]
Oct 31 02:00:00.545764 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.543741    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49"
Oct 31 02:00:00.546786 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.546761    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6"]
Oct 31 02:00:00.552408 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.552240    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87648fb-f2fb-42b8-b36b-22381fac5d7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg\" (UID: \"d87648fb-f2fb-42b8-b36b-22381fac5d7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg"
Oct 31 02:00:00.552408 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.552291    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb6mr\" (UniqueName: \"kubernetes.io/projected/d87648fb-f2fb-42b8-b36b-22381fac5d7d-kube-api-access-hb6mr\") pod \"maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg\" (UID: \"d87648fb-f2fb-42b8-b36b-22381fac5d7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg"
Oct 31 02:00:00.553515 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.553487    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5c76130-63ba-44f3-aa9e-c4ee0bceccec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw\" (UID: \"c5c76130-63ba-44f3-aa9e-c4ee0bceccec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw"
Oct 31 02:00:00.553627 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.553536    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c56489d-d1a0-4ae3-9260-eb0d18201d12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp\" (UID: \"0c56489d-d1a0-4ae3-9260-eb0d18201d12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp"
Oct 31 02:00:00.553627 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.553569    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvmvh\" (UniqueName: \"kubernetes.io/projected/0c56489d-d1a0-4ae3-9260-eb0d18201d12-kube-api-access-lvmvh\") pod \"maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp\" (UID: \"0c56489d-d1a0-4ae3-9260-eb0d18201d12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp"
Oct 31 02:00:00.553715 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.553685    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f86643f-3419-470f-a3e9-4aae61fc266c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6\" (UID: \"8f86643f-3419-470f-a3e9-4aae61fc266c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6"
Oct 31 02:00:00.554025 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.553803    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8xlv\" (UniqueName: \"kubernetes.io/projected/8f86643f-3419-470f-a3e9-4aae61fc266c-kube-api-access-n8xlv\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6\" (UID: \"8f86643f-3419-470f-a3e9-4aae61fc266c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6"
Oct 31 02:00:00.554025 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.553842    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgnmd\" (UniqueName: \"kubernetes.io/projected/e28001ed-ca1a-4790-96f5-73ada860e6de-kube-api-access-tgnmd\") pod \"maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2\" (UID: \"e28001ed-ca1a-4790-96f5-73ada860e6de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2"
Oct 31 02:00:00.554025 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.553910    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e28001ed-ca1a-4790-96f5-73ada860e6de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2\" (UID: \"e28001ed-ca1a-4790-96f5-73ada860e6de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2"
Oct 31 02:00:00.554025 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.553982    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j28xx\" (UniqueName: \"kubernetes.io/projected/c5c76130-63ba-44f3-aa9e-c4ee0bceccec-kube-api-access-j28xx\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw\" (UID: \"c5c76130-63ba-44f3-aa9e-c4ee0bceccec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw"
Oct 31 02:00:00.656465 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.656053    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7s9z\" (UniqueName: \"kubernetes.io/projected/b428ad4f-cd43-4daa-8f01-e94c7ef63732-kube-api-access-j7s9z\") pod \"maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf\" (UID: \"b428ad4f-cd43-4daa-8f01-e94c7ef63732\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf"
Oct 31 02:00:00.656465 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.656113    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c56489d-d1a0-4ae3-9260-eb0d18201d12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp\" (UID: \"0c56489d-d1a0-4ae3-9260-eb0d18201d12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp"
Oct 31 02:00:00.656465 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.656156    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvmvh\" (UniqueName: \"kubernetes.io/projected/0c56489d-d1a0-4ae3-9260-eb0d18201d12-kube-api-access-lvmvh\") pod \"maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp\" (UID: \"0c56489d-d1a0-4ae3-9260-eb0d18201d12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp"
Oct 31 02:00:00.656465 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.656223    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f86643f-3419-470f-a3e9-4aae61fc266c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6\" (UID: \"8f86643f-3419-470f-a3e9-4aae61fc266c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6"
Oct 31 02:00:00.656465 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.656297    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75n9f\" (UniqueName: \"kubernetes.io/projected/2bcc823c-fc5f-47c0-9cfa-665f05c71056-kube-api-access-75n9f\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh\" (UID: \"2bcc823c-fc5f-47c0-9cfa-665f05c71056\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh"
Oct 31 02:00:00.656465 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.656323    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9080245-f30d-4cb3-8c3a-61ea60ae2dd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw\" (UID: \"b9080245-f30d-4cb3-8c3a-61ea60ae2dd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw"
Oct 31 02:00:00.656465 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.656383    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bcc823c-fc5f-47c0-9cfa-665f05c71056-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh\" (UID: \"2bcc823c-fc5f-47c0-9cfa-665f05c71056\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh"
Oct 31 02:00:00.656465 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.656415    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8xlv\" (UniqueName: \"kubernetes.io/projected/8f86643f-3419-470f-a3e9-4aae61fc266c-kube-api-access-n8xlv\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6\" (UID: \"8f86643f-3419-470f-a3e9-4aae61fc266c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6"
Oct 31 02:00:00.656465 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.656466    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgnmd\" (UniqueName: \"kubernetes.io/projected/e28001ed-ca1a-4790-96f5-73ada860e6de-kube-api-access-tgnmd\") pod \"maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2\" (UID: \"e28001ed-ca1a-4790-96f5-73ada860e6de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2"
Oct 31 02:00:00.657089 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.656515    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e28001ed-ca1a-4790-96f5-73ada860e6de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2\" (UID: \"e28001ed-ca1a-4790-96f5-73ada860e6de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2"
Oct 31 02:00:00.657089 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.656567    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j28xx\" (UniqueName: \"kubernetes.io/projected/c5c76130-63ba-44f3-aa9e-c4ee0bceccec-kube-api-access-j28xx\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw\" (UID: \"c5c76130-63ba-44f3-aa9e-c4ee0bceccec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw"
Oct 31 02:00:00.657089 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.656628    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49\" (UID: \"c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49"
Oct 31 02:00:00.657089 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.656709    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87648fb-f2fb-42b8-b36b-22381fac5d7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg\" (UID: \"d87648fb-f2fb-42b8-b36b-22381fac5d7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg"
Oct 31 02:00:00.657089 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.656784    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb6mr\" (UniqueName: \"kubernetes.io/projected/d87648fb-f2fb-42b8-b36b-22381fac5d7d-kube-api-access-hb6mr\") pod \"maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg\" (UID: \"d87648fb-f2fb-42b8-b36b-22381fac5d7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg"
Oct 31 02:00:00.657089 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.656893    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b428ad4f-cd43-4daa-8f01-e94c7ef63732-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf\" (UID: \"b428ad4f-cd43-4daa-8f01-e94c7ef63732\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf"
Oct 31 02:00:00.658224 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.658054    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c56489d-d1a0-4ae3-9260-eb0d18201d12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp\" (UID: \"0c56489d-d1a0-4ae3-9260-eb0d18201d12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp"
Oct 31 02:00:00.658224 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.658195    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f86643f-3419-470f-a3e9-4aae61fc266c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6\" (UID: \"8f86643f-3419-470f-a3e9-4aae61fc266c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6"
Oct 31 02:00:00.658414 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.658395    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87648fb-f2fb-42b8-b36b-22381fac5d7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg\" (UID: \"d87648fb-f2fb-42b8-b36b-22381fac5d7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg"
Oct 31 02:00:00.658603 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.658584    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e28001ed-ca1a-4790-96f5-73ada860e6de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2\" (UID: \"e28001ed-ca1a-4790-96f5-73ada860e6de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2"
Oct 31 02:00:00.658692 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.658676    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfpk9\" (UniqueName: \"kubernetes.io/projected/c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d-kube-api-access-cfpk9\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49\" (UID: \"c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49"
Oct 31 02:00:00.658748 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.658736    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft2bk\" (UniqueName: \"kubernetes.io/projected/b9080245-f30d-4cb3-8c3a-61ea60ae2dd1-kube-api-access-ft2bk\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw\" (UID: \"b9080245-f30d-4cb3-8c3a-61ea60ae2dd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw"
Oct 31 02:00:00.658839 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.658820    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5c76130-63ba-44f3-aa9e-c4ee0bceccec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw\" (UID: \"c5c76130-63ba-44f3-aa9e-c4ee0bceccec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw"
Oct 31 02:00:00.659374 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.659344    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5c76130-63ba-44f3-aa9e-c4ee0bceccec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw\" (UID: \"c5c76130-63ba-44f3-aa9e-c4ee0bceccec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw"
Oct 31 02:00:00.675836 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.675781    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvmvh\" (UniqueName: \"kubernetes.io/projected/0c56489d-d1a0-4ae3-9260-eb0d18201d12-kube-api-access-lvmvh\") pod \"maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp\" (UID: \"0c56489d-d1a0-4ae3-9260-eb0d18201d12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp"
Oct 31 02:00:00.676956 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.676926    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb6mr\" (UniqueName: \"kubernetes.io/projected/d87648fb-f2fb-42b8-b36b-22381fac5d7d-kube-api-access-hb6mr\") pod \"maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg\" (UID: \"d87648fb-f2fb-42b8-b36b-22381fac5d7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg"
Oct 31 02:00:00.680278 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.680246    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j28xx\" (UniqueName: \"kubernetes.io/projected/c5c76130-63ba-44f3-aa9e-c4ee0bceccec-kube-api-access-j28xx\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw\" (UID: \"c5c76130-63ba-44f3-aa9e-c4ee0bceccec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw"
Oct 31 02:00:00.684471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.684356    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8xlv\" (UniqueName: \"kubernetes.io/projected/8f86643f-3419-470f-a3e9-4aae61fc266c-kube-api-access-n8xlv\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6\" (UID: \"8f86643f-3419-470f-a3e9-4aae61fc266c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6"
Oct 31 02:00:00.685244 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.685216    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgnmd\" (UniqueName: \"kubernetes.io/projected/e28001ed-ca1a-4790-96f5-73ada860e6de-kube-api-access-tgnmd\") pod \"maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2\" (UID: \"e28001ed-ca1a-4790-96f5-73ada860e6de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2"
Oct 31 02:00:00.760284 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.760243    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49\" (UID: \"c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49"
Oct 31 02:00:00.760525 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.760356    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b428ad4f-cd43-4daa-8f01-e94c7ef63732-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf\" (UID: \"b428ad4f-cd43-4daa-8f01-e94c7ef63732\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf"
Oct 31 02:00:00.760525 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.760394    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfpk9\" (UniqueName: \"kubernetes.io/projected/c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d-kube-api-access-cfpk9\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49\" (UID: \"c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49"
Oct 31 02:00:00.760525 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.760441    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft2bk\" (UniqueName: \"kubernetes.io/projected/b9080245-f30d-4cb3-8c3a-61ea60ae2dd1-kube-api-access-ft2bk\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw\" (UID: \"b9080245-f30d-4cb3-8c3a-61ea60ae2dd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw"
Oct 31 02:00:00.760525 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.760513    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7s9z\" (UniqueName: \"kubernetes.io/projected/b428ad4f-cd43-4daa-8f01-e94c7ef63732-kube-api-access-j7s9z\") pod \"maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf\" (UID: \"b428ad4f-cd43-4daa-8f01-e94c7ef63732\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf"
Oct 31 02:00:00.760789 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.760586    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75n9f\" (UniqueName: \"kubernetes.io/projected/2bcc823c-fc5f-47c0-9cfa-665f05c71056-kube-api-access-75n9f\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh\" (UID: \"2bcc823c-fc5f-47c0-9cfa-665f05c71056\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh"
Oct 31 02:00:00.760789 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.760612    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9080245-f30d-4cb3-8c3a-61ea60ae2dd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw\" (UID: \"b9080245-f30d-4cb3-8c3a-61ea60ae2dd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw"
Oct 31 02:00:00.760789 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.760657    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bcc823c-fc5f-47c0-9cfa-665f05c71056-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh\" (UID: \"2bcc823c-fc5f-47c0-9cfa-665f05c71056\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh"
Oct 31 02:00:00.761010 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.760987    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49\" (UID: \"c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49"
Oct 31 02:00:00.761282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.761257    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b428ad4f-cd43-4daa-8f01-e94c7ef63732-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf\" (UID: \"b428ad4f-cd43-4daa-8f01-e94c7ef63732\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf"
Oct 31 02:00:00.761435 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.761413    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bcc823c-fc5f-47c0-9cfa-665f05c71056-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh\" (UID: \"2bcc823c-fc5f-47c0-9cfa-665f05c71056\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh"
Oct 31 02:00:00.761516 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.761492    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9080245-f30d-4cb3-8c3a-61ea60ae2dd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw\" (UID: \"b9080245-f30d-4cb3-8c3a-61ea60ae2dd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw"
Oct 31 02:00:00.771607 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.771579    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw"
Oct 31 02:00:00.794974 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.776019    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft2bk\" (UniqueName: \"kubernetes.io/projected/b9080245-f30d-4cb3-8c3a-61ea60ae2dd1-kube-api-access-ft2bk\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw\" (UID: \"b9080245-f30d-4cb3-8c3a-61ea60ae2dd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw"
Oct 31 02:00:00.794974 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.776424    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75n9f\" (UniqueName: \"kubernetes.io/projected/2bcc823c-fc5f-47c0-9cfa-665f05c71056-kube-api-access-75n9f\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh\" (UID: \"2bcc823c-fc5f-47c0-9cfa-665f05c71056\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh"
Oct 31 02:00:00.812478 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.808747    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg"
Oct 31 02:00:00.847476 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.818265    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp"
Oct 31 02:00:00.856497 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.850913    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6"
Oct 31 02:00:00.856497 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.853047    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfpk9\" (UniqueName: \"kubernetes.io/projected/c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d-kube-api-access-cfpk9\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49\" (UID: \"c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49"
Oct 31 02:00:00.856497 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.853625    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7s9z\" (UniqueName: \"kubernetes.io/projected/b428ad4f-cd43-4daa-8f01-e94c7ef63732-kube-api-access-j7s9z\") pod \"maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf\" (UID: \"b428ad4f-cd43-4daa-8f01-e94c7ef63732\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf"
Oct 31 02:00:00.857868 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.857800    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2"
Oct 31 02:00:00.868433 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.868394    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw"
Oct 31 02:00:00.891482 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.890789    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh"
Oct 31 02:00:00.895329 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.895251    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf"
Oct 31 02:00:00.907586 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.906932    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 31 02:00:00.907586 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.906991    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn" podUID="4238ae64-8bce-4c8d-8f14-339706f25bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 31 02:00:00.915861 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:00.915836    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49"
Oct 31 02:00:01.678211 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:01.678008    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw"]
Oct 31 02:00:01.934256 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:01.934225    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 31 02:00:01.937727 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:01.934286    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn" podUID="4238ae64-8bce-4c8d-8f14-339706f25bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 31 02:00:02.158299 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:02.158259    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg"]
Oct 31 02:00:02.162966 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:00:02.162922    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87648fb_f2fb_42b8_b36b_22381fac5d7d.slice/crio-05ebfa2770f11f3d80a4bbf452a745d3cde7371856939881acfd77950f887851 WatchSource:0}: Error finding container 05ebfa2770f11f3d80a4bbf452a745d3cde7371856939881acfd77950f887851: Status 404 returned error can't find the container with id 05ebfa2770f11f3d80a4bbf452a745d3cde7371856939881acfd77950f887851
Oct 31 02:00:02.242946 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:02.242920    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw"]
Oct 31 02:00:02.248200 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:02.248139    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg" event={"ID":"d87648fb-f2fb-42b8-b36b-22381fac5d7d","Type":"ContainerStarted","Data":"05ebfa2770f11f3d80a4bbf452a745d3cde7371856939881acfd77950f887851"}
Oct 31 02:00:02.250610 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:02.250578    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw" event={"ID":"c5c76130-63ba-44f3-aa9e-c4ee0bceccec","Type":"ContainerStarted","Data":"4eead39c8bb3f651e1e90e37c6da3fd8739eaa8dbe757a543980ea7e172c50d6"}
Oct 31 02:00:02.255831 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:00:02.255723    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9080245_f30d_4cb3_8c3a_61ea60ae2dd1.slice/crio-df3a39723c252489c1327addef62d95c92afab0435e3b2087cb14b13755a9cb6 WatchSource:0}: Error finding container df3a39723c252489c1327addef62d95c92afab0435e3b2087cb14b13755a9cb6: Status 404 returned error can't find the container with id df3a39723c252489c1327addef62d95c92afab0435e3b2087cb14b13755a9cb6
Oct 31 02:00:02.269851 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:02.269618    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf"]
Oct 31 02:00:02.271306 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:00:02.271267    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb428ad4f_cd43_4daa_8f01_e94c7ef63732.slice/crio-b9ef2d4c310382cba83b758bc43044855841af22693a0b905ae08bc8772470f0 WatchSource:0}: Error finding container b9ef2d4c310382cba83b758bc43044855841af22693a0b905ae08bc8772470f0: Status 404 returned error can't find the container with id b9ef2d4c310382cba83b758bc43044855841af22693a0b905ae08bc8772470f0
Oct 31 02:00:02.283685 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:00:02.282969    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53f7ff7_14b2_4cbd_a7ba_0e848e2e7f1d.slice/crio-9885d03b54ea71ba1b86fdd5157468b46b5c18d8a5fc91907c5536be8be2b308 WatchSource:0}: Error finding container 9885d03b54ea71ba1b86fdd5157468b46b5c18d8a5fc91907c5536be8be2b308: Status 404 returned error can't find the container with id 9885d03b54ea71ba1b86fdd5157468b46b5c18d8a5fc91907c5536be8be2b308
Oct 31 02:00:02.285571 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:02.285500    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49"]
Oct 31 02:00:02.291553 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:00:02.291521    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f86643f_3419_470f_a3e9_4aae61fc266c.slice/crio-74b96405b086ab6f5ef920d4931bc0e39c12402b4e65ca773587261c806dc7af WatchSource:0}: Error finding container 74b96405b086ab6f5ef920d4931bc0e39c12402b4e65ca773587261c806dc7af: Status 404 returned error can't find the container with id 74b96405b086ab6f5ef920d4931bc0e39c12402b4e65ca773587261c806dc7af
Oct 31 02:00:02.293511 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:02.293487    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6"]
Oct 31 02:00:02.477457 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:02.475059    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh"]
Oct 31 02:00:02.542927 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:02.542895    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2"]
Oct 31 02:00:02.549743 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:00:02.549713    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c56489d_d1a0_4ae3_9260_eb0d18201d12.slice/crio-41f05c973ccd4c743bad6cdb67fac18a809d7751f704987dbbbc98448416dbc8 WatchSource:0}: Error finding container 41f05c973ccd4c743bad6cdb67fac18a809d7751f704987dbbbc98448416dbc8: Status 404 returned error can't find the container with id 41f05c973ccd4c743bad6cdb67fac18a809d7751f704987dbbbc98448416dbc8
Oct 31 02:00:02.553976 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:02.553945    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp"]
Oct 31 02:00:02.905923 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:02.905877    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 31 02:00:02.906187 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:02.906141    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn" podUID="4238ae64-8bce-4c8d-8f14-339706f25bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 31 02:00:02.923236 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:02.923204    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 02:00:03.261670 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:03.261630    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg" event={"ID":"d87648fb-f2fb-42b8-b36b-22381fac5d7d","Type":"ContainerStarted","Data":"464e3034727f9beaba475a4eb00da616fe964aad809b695128bee8cd321d464b"}
Oct 31 02:00:03.263857 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:03.263820    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw" event={"ID":"b9080245-f30d-4cb3-8c3a-61ea60ae2dd1","Type":"ContainerStarted","Data":"bbc5a095c3d4bfdeb8aeb487ddc108c9b82d4fada524e1d6a1a24b63d0ffbe3d"}
Oct 31 02:00:03.264027 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:03.263877    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw" event={"ID":"b9080245-f30d-4cb3-8c3a-61ea60ae2dd1","Type":"ContainerStarted","Data":"df3a39723c252489c1327addef62d95c92afab0435e3b2087cb14b13755a9cb6"}
Oct 31 02:00:03.265315 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:03.265288    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2" event={"ID":"e28001ed-ca1a-4790-96f5-73ada860e6de","Type":"ContainerStarted","Data":"bf51f391ba7b66c29d95c62155255765ef64c0f02e14bbc84da2f1847347dbbd"}
Oct 31 02:00:03.267295 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:03.267265    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf" event={"ID":"b428ad4f-cd43-4daa-8f01-e94c7ef63732","Type":"ContainerStarted","Data":"5307e12d1eaab65ea96089fc3afc9bd07b89043f210f0f50efc04326c2366d95"}
Oct 31 02:00:03.267429 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:03.267309    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf" event={"ID":"b428ad4f-cd43-4daa-8f01-e94c7ef63732","Type":"ContainerStarted","Data":"b9ef2d4c310382cba83b758bc43044855841af22693a0b905ae08bc8772470f0"}
Oct 31 02:00:03.269232 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:03.269208    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6" event={"ID":"8f86643f-3419-470f-a3e9-4aae61fc266c","Type":"ContainerStarted","Data":"f5e0236b8b2d3b1fc5b083a2f021289833c01ffd56bd31276c6064969b318413"}
Oct 31 02:00:03.269357 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:03.269251    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6" event={"ID":"8f86643f-3419-470f-a3e9-4aae61fc266c","Type":"ContainerStarted","Data":"74b96405b086ab6f5ef920d4931bc0e39c12402b4e65ca773587261c806dc7af"}
Oct 31 02:00:03.270873 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:03.270850    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp" event={"ID":"0c56489d-d1a0-4ae3-9260-eb0d18201d12","Type":"ContainerStarted","Data":"41f05c973ccd4c743bad6cdb67fac18a809d7751f704987dbbbc98448416dbc8"}
Oct 31 02:00:03.272672 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:03.272641    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw" event={"ID":"c5c76130-63ba-44f3-aa9e-c4ee0bceccec","Type":"ContainerStarted","Data":"670b77834e2d369438b4acb598482be17e9241d06186c3d5573d56422756c156"}
Oct 31 02:00:03.274386 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:03.274362    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49" event={"ID":"c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d","Type":"ContainerStarted","Data":"d17b932194fe257879e7ccca8de95a236472e135ec176a1ad8b06ef92c1cef16"}
Oct 31 02:00:03.274503 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:03.274405    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49" event={"ID":"c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d","Type":"ContainerStarted","Data":"9885d03b54ea71ba1b86fdd5157468b46b5c18d8a5fc91907c5536be8be2b308"}
Oct 31 02:00:03.276244 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:03.276222    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh" event={"ID":"2bcc823c-fc5f-47c0-9cfa-665f05c71056","Type":"ContainerStarted","Data":"f4cc6aecb51fb78c7e35a6d57a9e8d324d06816ce51ac65c267fc690ef7e13ab"}
Oct 31 02:00:03.276359 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:03.276264    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh" event={"ID":"2bcc823c-fc5f-47c0-9cfa-665f05c71056","Type":"ContainerStarted","Data":"b9d3d3f06db8f2d7033b9b476ba4833b3f47c60252295c9fa62558dede767d59"}
Oct 31 02:00:03.285756 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:03.285691    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg" podStartSLOduration=3.28566854 podStartE2EDuration="3.28566854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:03.281224328 +0000 UTC m=+2359.284938620" watchObservedRunningTime="2026-10-31 02:00:03.28566854 +0000 UTC m=+2359.289382832"
Oct 31 02:00:03.306621 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:03.306558    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw" podStartSLOduration=3.306539944 podStartE2EDuration="3.306539944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:03.302093539 +0000 UTC m=+2359.305807829" watchObservedRunningTime="2026-10-31 02:00:03.306539944 +0000 UTC m=+2359.310254236"
Oct 31 02:00:03.328118 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:03.328045    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49" podStartSLOduration=3.328021564 podStartE2EDuration="3.328021564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:03.32610739 +0000 UTC m=+2359.329821683" watchObservedRunningTime="2026-10-31 02:00:03.328021564 +0000 UTC m=+2359.331735856"
Oct 31 02:00:03.353983 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:03.353911    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh" podStartSLOduration=3.353894137 podStartE2EDuration="3.353894137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:03.34898895 +0000 UTC m=+2359.352703242" watchObservedRunningTime="2026-10-31 02:00:03.353894137 +0000 UTC m=+2359.357608428"
Oct 31 02:00:03.376478 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:03.376355    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf" podStartSLOduration=3.376334164 podStartE2EDuration="3.376334164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:03.37224013 +0000 UTC m=+2359.375954436" watchObservedRunningTime="2026-10-31 02:00:03.376334164 +0000 UTC m=+2359.380048456"
Oct 31 02:00:03.397615 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:03.397531    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6" podStartSLOduration=3.39751427 podStartE2EDuration="3.39751427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:03.395836381 +0000 UTC m=+2359.399550673" watchObservedRunningTime="2026-10-31 02:00:03.39751427 +0000 UTC m=+2359.401228560"
Oct 31 02:00:03.433981 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:03.433925    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw" podStartSLOduration=3.433909613 podStartE2EDuration="3.433909613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:03.428828792 +0000 UTC m=+2359.432543074" watchObservedRunningTime="2026-10-31 02:00:03.433909613 +0000 UTC m=+2359.437623904"
Oct 31 02:00:03.910611 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:03.910580    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 02:00:03.912156 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:03.912128    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 02:00:04.289231 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.289190    5650 generic.go:356] "Generic (PLEG): container finished" podID="0c56489d-d1a0-4ae3-9260-eb0d18201d12" containerID="6a99bb78566e1494f3b7534017f952e2e1b23096d9342d28c55f32eda2024cab" exitCode=60
Oct 31 02:00:04.289666 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.289247    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp" event={"ID":"0c56489d-d1a0-4ae3-9260-eb0d18201d12","Type":"ContainerDied","Data":"6a99bb78566e1494f3b7534017f952e2e1b23096d9342d28c55f32eda2024cab"}
Oct 31 02:00:04.289764 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.289748    5650 scope.go:122] "RemoveContainer" containerID="6a99bb78566e1494f3b7534017f952e2e1b23096d9342d28c55f32eda2024cab"
Oct 31 02:00:04.291502 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.291470    5650 generic.go:356] "Generic (PLEG): container finished" podID="c5c76130-63ba-44f3-aa9e-c4ee0bceccec" containerID="670b77834e2d369438b4acb598482be17e9241d06186c3d5573d56422756c156" exitCode=60
Oct 31 02:00:04.291648 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.291611    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw" event={"ID":"c5c76130-63ba-44f3-aa9e-c4ee0bceccec","Type":"ContainerDied","Data":"670b77834e2d369438b4acb598482be17e9241d06186c3d5573d56422756c156"}
Oct 31 02:00:04.292337 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.292220    5650 scope.go:122] "RemoveContainer" containerID="670b77834e2d369438b4acb598482be17e9241d06186c3d5573d56422756c156"
Oct 31 02:00:04.293742 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.293712    5650 generic.go:356] "Generic (PLEG): container finished" podID="c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d" containerID="d17b932194fe257879e7ccca8de95a236472e135ec176a1ad8b06ef92c1cef16" exitCode=60
Oct 31 02:00:04.293880 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.293820    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49" event={"ID":"c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d","Type":"ContainerDied","Data":"d17b932194fe257879e7ccca8de95a236472e135ec176a1ad8b06ef92c1cef16"}
Oct 31 02:00:04.294187 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.294168    5650 scope.go:122] "RemoveContainer" containerID="d17b932194fe257879e7ccca8de95a236472e135ec176a1ad8b06ef92c1cef16"
Oct 31 02:00:04.299303 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.299279    5650 generic.go:356] "Generic (PLEG): container finished" podID="2bcc823c-fc5f-47c0-9cfa-665f05c71056" containerID="f4cc6aecb51fb78c7e35a6d57a9e8d324d06816ce51ac65c267fc690ef7e13ab" exitCode=60
Oct 31 02:00:04.299435 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.299361    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh" event={"ID":"2bcc823c-fc5f-47c0-9cfa-665f05c71056","Type":"ContainerDied","Data":"f4cc6aecb51fb78c7e35a6d57a9e8d324d06816ce51ac65c267fc690ef7e13ab"}
Oct 31 02:00:04.299782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.299758    5650 scope.go:122] "RemoveContainer" containerID="f4cc6aecb51fb78c7e35a6d57a9e8d324d06816ce51ac65c267fc690ef7e13ab"
Oct 31 02:00:04.301227 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.301207    5650 generic.go:356] "Generic (PLEG): container finished" podID="d87648fb-f2fb-42b8-b36b-22381fac5d7d" containerID="464e3034727f9beaba475a4eb00da616fe964aad809b695128bee8cd321d464b" exitCode=60
Oct 31 02:00:04.301353 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.301331    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg" event={"ID":"d87648fb-f2fb-42b8-b36b-22381fac5d7d","Type":"ContainerDied","Data":"464e3034727f9beaba475a4eb00da616fe964aad809b695128bee8cd321d464b"}
Oct 31 02:00:04.301788 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.301768    5650 scope.go:122] "RemoveContainer" containerID="464e3034727f9beaba475a4eb00da616fe964aad809b695128bee8cd321d464b"
Oct 31 02:00:04.303936 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.303910    5650 generic.go:356] "Generic (PLEG): container finished" podID="b9080245-f30d-4cb3-8c3a-61ea60ae2dd1" containerID="bbc5a095c3d4bfdeb8aeb487ddc108c9b82d4fada524e1d6a1a24b63d0ffbe3d" exitCode=60
Oct 31 02:00:04.304865 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.304163    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw" event={"ID":"b9080245-f30d-4cb3-8c3a-61ea60ae2dd1","Type":"ContainerDied","Data":"bbc5a095c3d4bfdeb8aeb487ddc108c9b82d4fada524e1d6a1a24b63d0ffbe3d"}
Oct 31 02:00:04.304865 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.304372    5650 scope.go:122] "RemoveContainer" containerID="bbc5a095c3d4bfdeb8aeb487ddc108c9b82d4fada524e1d6a1a24b63d0ffbe3d"
Oct 31 02:00:04.306193 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.306171    5650 generic.go:356] "Generic (PLEG): container finished" podID="e28001ed-ca1a-4790-96f5-73ada860e6de" containerID="be117a8219afbe4e05375ee86a4ed00b5589e36619bb9db7dc91edc655ad0de6" exitCode=60
Oct 31 02:00:04.306286 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.306249    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2" event={"ID":"e28001ed-ca1a-4790-96f5-73ada860e6de","Type":"ContainerDied","Data":"be117a8219afbe4e05375ee86a4ed00b5589e36619bb9db7dc91edc655ad0de6"}
Oct 31 02:00:04.306523 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.306504    5650 scope.go:122] "RemoveContainer" containerID="be117a8219afbe4e05375ee86a4ed00b5589e36619bb9db7dc91edc655ad0de6"
Oct 31 02:00:04.308626 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.308604    5650 generic.go:356] "Generic (PLEG): container finished" podID="b428ad4f-cd43-4daa-8f01-e94c7ef63732" containerID="5307e12d1eaab65ea96089fc3afc9bd07b89043f210f0f50efc04326c2366d95" exitCode=60
Oct 31 02:00:04.309027 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.308849    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf" event={"ID":"b428ad4f-cd43-4daa-8f01-e94c7ef63732","Type":"ContainerDied","Data":"5307e12d1eaab65ea96089fc3afc9bd07b89043f210f0f50efc04326c2366d95"}
Oct 31 02:00:04.310872 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.310850    5650 scope.go:122] "RemoveContainer" containerID="5307e12d1eaab65ea96089fc3afc9bd07b89043f210f0f50efc04326c2366d95"
Oct 31 02:00:04.311167 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.311069    5650 generic.go:356] "Generic (PLEG): container finished" podID="8f86643f-3419-470f-a3e9-4aae61fc266c" containerID="f5e0236b8b2d3b1fc5b083a2f021289833c01ffd56bd31276c6064969b318413" exitCode=60
Oct 31 02:00:04.311291 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.311259    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6" event={"ID":"8f86643f-3419-470f-a3e9-4aae61fc266c","Type":"ContainerDied","Data":"f5e0236b8b2d3b1fc5b083a2f021289833c01ffd56bd31276c6064969b318413"}
Oct 31 02:00:04.311691 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:04.311670    5650 scope.go:122] "RemoveContainer" containerID="f5e0236b8b2d3b1fc5b083a2f021289833c01ffd56bd31276c6064969b318413"
Oct 31 02:00:05.323396 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:05.323359    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg" event={"ID":"d87648fb-f2fb-42b8-b36b-22381fac5d7d","Type":"ContainerStarted","Data":"3777de98872d2de1506658778cbe8dfd096767e00d3647e08b6b1c4328e80520"}
Oct 31 02:00:05.329376 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:05.329350    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf" event={"ID":"b428ad4f-cd43-4daa-8f01-e94c7ef63732","Type":"ContainerStarted","Data":"ba8301509b47f9a815523adbc6e2df712c296f8c9ee67dc06a0945fc8d8233dc"}
Oct 31 02:00:05.331368 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:05.331342    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6" event={"ID":"8f86643f-3419-470f-a3e9-4aae61fc266c","Type":"ContainerStarted","Data":"1229c47c87eb61bf0851bb7a17eca95625526272d0b5435e31ea8d16cb719d6e"}
Oct 31 02:00:05.333289 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:05.333264    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp" event={"ID":"0c56489d-d1a0-4ae3-9260-eb0d18201d12","Type":"ContainerStarted","Data":"bc833d52334c40098a6aa4bfb295a84ace899d27cf5568bb7801d7848119684c"}
Oct 31 02:00:05.335670 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:05.335641    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw" event={"ID":"c5c76130-63ba-44f3-aa9e-c4ee0bceccec","Type":"ContainerStarted","Data":"e8df67abaa157b23d09be95d88cf22855209316ab7d8b4f8983b9b998b8ad35a"}
Oct 31 02:00:05.337640 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:05.337607    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49" event={"ID":"c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d","Type":"ContainerStarted","Data":"c456e81dd3a67bbe10e6610957a6816922da32a9aa56642a9b4d6dc78e3fbf84"}
Oct 31 02:00:05.339558 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:05.339537    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh" event={"ID":"2bcc823c-fc5f-47c0-9cfa-665f05c71056","Type":"ContainerStarted","Data":"c3d2d3dcc6e42b4884f39939ff10101b04693995ce4ebd06951e68a20474b4ec"}
Oct 31 02:00:05.375029 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:05.374978    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2" podStartSLOduration=5.374963322 podStartE2EDuration="5.374963322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:05.372410677 +0000 UTC m=+2361.376124968" watchObservedRunningTime="2026-10-31 02:00:05.374963322 +0000 UTC m=+2361.378677613"
Oct 31 02:00:05.396370 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:05.396287    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp" podStartSLOduration=5.396268846 podStartE2EDuration="5.396268846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:05.394604445 +0000 UTC m=+2361.398318740" watchObservedRunningTime="2026-10-31 02:00:05.396268846 +0000 UTC m=+2361.399983136"
Oct 31 02:00:06.351518 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.351479    5650 generic.go:356] "Generic (PLEG): container finished" podID="d87648fb-f2fb-42b8-b36b-22381fac5d7d" containerID="3777de98872d2de1506658778cbe8dfd096767e00d3647e08b6b1c4328e80520" exitCode=60
Oct 31 02:00:06.352094 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.351525    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg" event={"ID":"d87648fb-f2fb-42b8-b36b-22381fac5d7d","Type":"ContainerDied","Data":"3777de98872d2de1506658778cbe8dfd096767e00d3647e08b6b1c4328e80520"}
Oct 31 02:00:06.352094 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.351601    5650 scope.go:122] "RemoveContainer" containerID="464e3034727f9beaba475a4eb00da616fe964aad809b695128bee8cd321d464b"
Oct 31 02:00:06.352094 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.352046    5650 scope.go:122] "RemoveContainer" containerID="3777de98872d2de1506658778cbe8dfd096767e00d3647e08b6b1c4328e80520"
Oct 31 02:00:06.354245 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.354215    5650 generic.go:356] "Generic (PLEG): container finished" podID="b9080245-f30d-4cb3-8c3a-61ea60ae2dd1" containerID="afe0af3bef6499d0363dccc5eb7e55706522540a17ac993c650e6d3d220c7a75" exitCode=60
Oct 31 02:00:06.354418 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.354344    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw" event={"ID":"b9080245-f30d-4cb3-8c3a-61ea60ae2dd1","Type":"ContainerDied","Data":"afe0af3bef6499d0363dccc5eb7e55706522540a17ac993c650e6d3d220c7a75"}
Oct 31 02:00:06.354876 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.354851    5650 scope.go:122] "RemoveContainer" containerID="afe0af3bef6499d0363dccc5eb7e55706522540a17ac993c650e6d3d220c7a75"
Oct 31 02:00:06.355263 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:00:06.355244    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw_odh-ai-gateway-infra(b9080245-f30d-4cb3-8c3a-61ea60ae2dd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw" podUID="b9080245-f30d-4cb3-8c3a-61ea60ae2dd1"
Oct 31 02:00:06.357417 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:00:06.357386    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg_odh-ai-gateway-infra(d87648fb-f2fb-42b8-b36b-22381fac5d7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg" podUID="d87648fb-f2fb-42b8-b36b-22381fac5d7d"
Oct 31 02:00:06.357766 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.357740    5650 generic.go:356] "Generic (PLEG): container finished" podID="e28001ed-ca1a-4790-96f5-73ada860e6de" containerID="398ff41d6ac5fab8e3c6e0ffb7136a036a53b98a590d9f1464b4216faee951f5" exitCode=60
Oct 31 02:00:06.357905 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.357766    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2" event={"ID":"e28001ed-ca1a-4790-96f5-73ada860e6de","Type":"ContainerDied","Data":"398ff41d6ac5fab8e3c6e0ffb7136a036a53b98a590d9f1464b4216faee951f5"}
Oct 31 02:00:06.359807 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.358260    5650 scope.go:122] "RemoveContainer" containerID="398ff41d6ac5fab8e3c6e0ffb7136a036a53b98a590d9f1464b4216faee951f5"
Oct 31 02:00:06.359807 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:00:06.358705    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2_odh-ai-gateway-infra(e28001ed-ca1a-4790-96f5-73ada860e6de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2" podUID="e28001ed-ca1a-4790-96f5-73ada860e6de"
Oct 31 02:00:06.360959 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.360924    5650 generic.go:356] "Generic (PLEG): container finished" podID="b428ad4f-cd43-4daa-8f01-e94c7ef63732" containerID="ba8301509b47f9a815523adbc6e2df712c296f8c9ee67dc06a0945fc8d8233dc" exitCode=60
Oct 31 02:00:06.361103 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.361077    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf" event={"ID":"b428ad4f-cd43-4daa-8f01-e94c7ef63732","Type":"ContainerDied","Data":"ba8301509b47f9a815523adbc6e2df712c296f8c9ee67dc06a0945fc8d8233dc"}
Oct 31 02:00:06.361378 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.361359    5650 scope.go:122] "RemoveContainer" containerID="ba8301509b47f9a815523adbc6e2df712c296f8c9ee67dc06a0945fc8d8233dc"
Oct 31 02:00:06.361924 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:00:06.361896    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf_odh-ai-gateway-infra(b428ad4f-cd43-4daa-8f01-e94c7ef63732)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf" podUID="b428ad4f-cd43-4daa-8f01-e94c7ef63732"
Oct 31 02:00:06.363957 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.363932    5650 generic.go:356] "Generic (PLEG): container finished" podID="8f86643f-3419-470f-a3e9-4aae61fc266c" containerID="1229c47c87eb61bf0851bb7a17eca95625526272d0b5435e31ea8d16cb719d6e" exitCode=60
Oct 31 02:00:06.364094 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.364032    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6" event={"ID":"8f86643f-3419-470f-a3e9-4aae61fc266c","Type":"ContainerDied","Data":"1229c47c87eb61bf0851bb7a17eca95625526272d0b5435e31ea8d16cb719d6e"}
Oct 31 02:00:06.364338 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.364322    5650 scope.go:122] "RemoveContainer" containerID="1229c47c87eb61bf0851bb7a17eca95625526272d0b5435e31ea8d16cb719d6e"
Oct 31 02:00:06.364735 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:00:06.364715    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6_odh-ai-gateway-infra(8f86643f-3419-470f-a3e9-4aae61fc266c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6" podUID="8f86643f-3419-470f-a3e9-4aae61fc266c"
Oct 31 02:00:06.367000 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.366972    5650 generic.go:356] "Generic (PLEG): container finished" podID="0c56489d-d1a0-4ae3-9260-eb0d18201d12" containerID="bc833d52334c40098a6aa4bfb295a84ace899d27cf5568bb7801d7848119684c" exitCode=60
Oct 31 02:00:06.367137 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.367014    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp" event={"ID":"0c56489d-d1a0-4ae3-9260-eb0d18201d12","Type":"ContainerDied","Data":"bc833d52334c40098a6aa4bfb295a84ace899d27cf5568bb7801d7848119684c"}
Oct 31 02:00:06.367516 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.367493    5650 scope.go:122] "RemoveContainer" containerID="bc833d52334c40098a6aa4bfb295a84ace899d27cf5568bb7801d7848119684c"
Oct 31 02:00:06.368103 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:00:06.368052    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp_odh-ai-gateway-infra(0c56489d-d1a0-4ae3-9260-eb0d18201d12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp" podUID="0c56489d-d1a0-4ae3-9260-eb0d18201d12"
Oct 31 02:00:06.369840 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.369810    5650 generic.go:356] "Generic (PLEG): container finished" podID="c5c76130-63ba-44f3-aa9e-c4ee0bceccec" containerID="e8df67abaa157b23d09be95d88cf22855209316ab7d8b4f8983b9b998b8ad35a" exitCode=60
Oct 31 02:00:06.369986 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.369896    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw" event={"ID":"c5c76130-63ba-44f3-aa9e-c4ee0bceccec","Type":"ContainerDied","Data":"e8df67abaa157b23d09be95d88cf22855209316ab7d8b4f8983b9b998b8ad35a"}
Oct 31 02:00:06.370480 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.370441    5650 scope.go:122] "RemoveContainer" containerID="e8df67abaa157b23d09be95d88cf22855209316ab7d8b4f8983b9b998b8ad35a"
Oct 31 02:00:06.371052 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:00:06.371024    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw_odh-ai-gateway-infra(c5c76130-63ba-44f3-aa9e-c4ee0bceccec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw" podUID="c5c76130-63ba-44f3-aa9e-c4ee0bceccec"
Oct 31 02:00:06.372370 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.372352    5650 scope.go:122] "RemoveContainer" containerID="bbc5a095c3d4bfdeb8aeb487ddc108c9b82d4fada524e1d6a1a24b63d0ffbe3d"
Oct 31 02:00:06.377247 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.377208    5650 generic.go:356] "Generic (PLEG): container finished" podID="c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d" containerID="c456e81dd3a67bbe10e6610957a6816922da32a9aa56642a9b4d6dc78e3fbf84" exitCode=60
Oct 31 02:00:06.377390 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.377295    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49" event={"ID":"c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d","Type":"ContainerDied","Data":"c456e81dd3a67bbe10e6610957a6816922da32a9aa56642a9b4d6dc78e3fbf84"}
Oct 31 02:00:06.377891 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.377865    5650 scope.go:122] "RemoveContainer" containerID="c456e81dd3a67bbe10e6610957a6816922da32a9aa56642a9b4d6dc78e3fbf84"
Oct 31 02:00:06.378805 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:00:06.378774    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49_odh-ai-gateway-infra(c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49" podUID="c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d"
Oct 31 02:00:06.379929 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.379905    5650 generic.go:356] "Generic (PLEG): container finished" podID="2bcc823c-fc5f-47c0-9cfa-665f05c71056" containerID="c3d2d3dcc6e42b4884f39939ff10101b04693995ce4ebd06951e68a20474b4ec" exitCode=60
Oct 31 02:00:06.380074 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.379947    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh" event={"ID":"2bcc823c-fc5f-47c0-9cfa-665f05c71056","Type":"ContainerDied","Data":"c3d2d3dcc6e42b4884f39939ff10101b04693995ce4ebd06951e68a20474b4ec"}
Oct 31 02:00:06.380545 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.380519    5650 scope.go:122] "RemoveContainer" containerID="c3d2d3dcc6e42b4884f39939ff10101b04693995ce4ebd06951e68a20474b4ec"
Oct 31 02:00:06.381112 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:00:06.381082    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh_odh-ai-gateway-infra(2bcc823c-fc5f-47c0-9cfa-665f05c71056)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh" podUID="2bcc823c-fc5f-47c0-9cfa-665f05c71056"
Oct 31 02:00:06.393203 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.393180    5650 scope.go:122] "RemoveContainer" containerID="be117a8219afbe4e05375ee86a4ed00b5589e36619bb9db7dc91edc655ad0de6"
Oct 31 02:00:06.411794 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.411767    5650 scope.go:122] "RemoveContainer" containerID="5307e12d1eaab65ea96089fc3afc9bd07b89043f210f0f50efc04326c2366d95"
Oct 31 02:00:06.430066 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.430044    5650 scope.go:122] "RemoveContainer" containerID="f5e0236b8b2d3b1fc5b083a2f021289833c01ffd56bd31276c6064969b318413"
Oct 31 02:00:06.447194 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.447158    5650 scope.go:122] "RemoveContainer" containerID="6a99bb78566e1494f3b7534017f952e2e1b23096d9342d28c55f32eda2024cab"
Oct 31 02:00:06.469400 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.469340    5650 scope.go:122] "RemoveContainer" containerID="670b77834e2d369438b4acb598482be17e9241d06186c3d5573d56422756c156"
Oct 31 02:00:06.491467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.491430    5650 scope.go:122] "RemoveContainer" containerID="d17b932194fe257879e7ccca8de95a236472e135ec176a1ad8b06ef92c1cef16"
Oct 31 02:00:06.508097 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:06.508077    5650 scope.go:122] "RemoveContainer" containerID="f4cc6aecb51fb78c7e35a6d57a9e8d324d06816ce51ac65c267fc690ef7e13ab"
Oct 31 02:00:07.391118 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:07.391090    5650 scope.go:122] "RemoveContainer" containerID="e8df67abaa157b23d09be95d88cf22855209316ab7d8b4f8983b9b998b8ad35a"
Oct 31 02:00:07.391729 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:00:07.391591    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw_odh-ai-gateway-infra(c5c76130-63ba-44f3-aa9e-c4ee0bceccec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw" podUID="c5c76130-63ba-44f3-aa9e-c4ee0bceccec"
Oct 31 02:00:07.393124 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:07.393093    5650 scope.go:122] "RemoveContainer" containerID="c456e81dd3a67bbe10e6610957a6816922da32a9aa56642a9b4d6dc78e3fbf84"
Oct 31 02:00:07.393588 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:00:07.393556    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49_odh-ai-gateway-infra(c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49" podUID="c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d"
Oct 31 02:00:07.395137 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:07.395114    5650 scope.go:122] "RemoveContainer" containerID="c3d2d3dcc6e42b4884f39939ff10101b04693995ce4ebd06951e68a20474b4ec"
Oct 31 02:00:07.395536 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:00:07.395505    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh_odh-ai-gateway-infra(2bcc823c-fc5f-47c0-9cfa-665f05c71056)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh" podUID="2bcc823c-fc5f-47c0-9cfa-665f05c71056"
Oct 31 02:00:07.397048 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:07.397025    5650 scope.go:122] "RemoveContainer" containerID="3777de98872d2de1506658778cbe8dfd096767e00d3647e08b6b1c4328e80520"
Oct 31 02:00:07.397441 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:00:07.397418    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg_odh-ai-gateway-infra(d87648fb-f2fb-42b8-b36b-22381fac5d7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg" podUID="d87648fb-f2fb-42b8-b36b-22381fac5d7d"
Oct 31 02:00:07.402290 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:07.402266    5650 scope.go:122] "RemoveContainer" containerID="ba8301509b47f9a815523adbc6e2df712c296f8c9ee67dc06a0945fc8d8233dc"
Oct 31 02:00:07.402677 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:00:07.402651    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf_odh-ai-gateway-infra(b428ad4f-cd43-4daa-8f01-e94c7ef63732)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf" podUID="b428ad4f-cd43-4daa-8f01-e94c7ef63732"
Oct 31 02:00:07.403943 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:07.403921    5650 scope.go:122] "RemoveContainer" containerID="1229c47c87eb61bf0851bb7a17eca95625526272d0b5435e31ea8d16cb719d6e"
Oct 31 02:00:07.404298 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:00:07.404274    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6_odh-ai-gateway-infra(8f86643f-3419-470f-a3e9-4aae61fc266c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6" podUID="8f86643f-3419-470f-a3e9-4aae61fc266c"
Oct 31 02:00:07.405571 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:07.405549    5650 scope.go:122] "RemoveContainer" containerID="bc833d52334c40098a6aa4bfb295a84ace899d27cf5568bb7801d7848119684c"
Oct 31 02:00:07.405869 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:00:07.405849    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp_odh-ai-gateway-infra(0c56489d-d1a0-4ae3-9260-eb0d18201d12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp" podUID="0c56489d-d1a0-4ae3-9260-eb0d18201d12"
Oct 31 02:00:15.371834 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:15.371795    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c"]
Oct 31 02:00:15.377614 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:15.377578    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c"
Oct 31 02:00:15.385115 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:15.385082    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c"]
Oct 31 02:00:15.391681 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:15.391648    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-d0c6ff-dockercfg-4sqw7\""
Oct 31 02:00:15.438928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:15.438884    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-d0c6ff\""
Oct 31 02:00:15.533517 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:15.507937    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcf95\" (UniqueName: \"kubernetes.io/projected/822c5afc-7c59-422d-a24d-d1b682127173-kube-api-access-gcf95\") pod \"payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c\" (UID: \"822c5afc-7c59-422d-a24d-d1b682127173\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c"
Oct 31 02:00:15.533517 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:15.508064    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/822c5afc-7c59-422d-a24d-d1b682127173-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c\" (UID: \"822c5afc-7c59-422d-a24d-d1b682127173\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c"
Oct 31 02:00:15.700866 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:15.700772    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd"]
Oct 31 02:00:15.772466 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:15.736737    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd"
Oct 31 02:00:15.828737 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:15.828682    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6eae8ee-0ae3-4beb-9777-e63e8853a9cb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd\" (UID: \"f6eae8ee-0ae3-4beb-9777-e63e8853a9cb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd"
Oct 31 02:00:15.854841 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:15.854794    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-859qt\" (UniqueName: \"kubernetes.io/projected/f6eae8ee-0ae3-4beb-9777-e63e8853a9cb-kube-api-access-859qt\") pod \"payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd\" (UID: \"f6eae8ee-0ae3-4beb-9777-e63e8853a9cb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd"
Oct 31 02:00:15.864028 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:15.863981    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcf95\" (UniqueName: \"kubernetes.io/projected/822c5afc-7c59-422d-a24d-d1b682127173-kube-api-access-gcf95\") pod \"payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c\" (UID: \"822c5afc-7c59-422d-a24d-d1b682127173\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c"
Oct 31 02:00:15.890617 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:15.886427    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/822c5afc-7c59-422d-a24d-d1b682127173-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c\" (UID: \"822c5afc-7c59-422d-a24d-d1b682127173\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c"
Oct 31 02:00:15.933471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:15.894095    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/822c5afc-7c59-422d-a24d-d1b682127173-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c\" (UID: \"822c5afc-7c59-422d-a24d-d1b682127173\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c"
Oct 31 02:00:15.959541 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:15.952922    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcf95\" (UniqueName: \"kubernetes.io/projected/822c5afc-7c59-422d-a24d-d1b682127173-kube-api-access-gcf95\") pod \"payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c\" (UID: \"822c5afc-7c59-422d-a24d-d1b682127173\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c"
Oct 31 02:00:15.978933 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:15.978794    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd"]
Oct 31 02:00:15.988473 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:15.988407    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-859qt\" (UniqueName: \"kubernetes.io/projected/f6eae8ee-0ae3-4beb-9777-e63e8853a9cb-kube-api-access-859qt\") pod \"payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd\" (UID: \"f6eae8ee-0ae3-4beb-9777-e63e8853a9cb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd"
Oct 31 02:00:15.989053 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:15.989023    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6eae8ee-0ae3-4beb-9777-e63e8853a9cb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd\" (UID: \"f6eae8ee-0ae3-4beb-9777-e63e8853a9cb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd"
Oct 31 02:00:15.990312 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:15.990288    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6eae8ee-0ae3-4beb-9777-e63e8853a9cb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd\" (UID: \"f6eae8ee-0ae3-4beb-9777-e63e8853a9cb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd"
Oct 31 02:00:15.999342 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:15.999306    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c"
Oct 31 02:00:16.019587 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:16.019555    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-859qt\" (UniqueName: \"kubernetes.io/projected/f6eae8ee-0ae3-4beb-9777-e63e8853a9cb-kube-api-access-859qt\") pod \"payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd\" (UID: \"f6eae8ee-0ae3-4beb-9777-e63e8853a9cb\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd"
Oct 31 02:00:16.132262 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:16.131621    5650 scope.go:122] "RemoveContainer" containerID="398ff41d6ac5fab8e3c6e0ffb7136a036a53b98a590d9f1464b4216faee951f5"
Oct 31 02:00:16.261236 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:16.261200    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd"
Oct 31 02:00:16.374174 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:00:16.374140    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822c5afc_7c59_422d_a24d_d1b682127173.slice/crio-040058f7441110dfac0c2c0ab9d91346e4f592cdcdafff125ddca6192e0f7883 WatchSource:0}: Error finding container 040058f7441110dfac0c2c0ab9d91346e4f592cdcdafff125ddca6192e0f7883: Status 404 returned error can't find the container with id 040058f7441110dfac0c2c0ab9d91346e4f592cdcdafff125ddca6192e0f7883
Oct 31 02:00:16.376410 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:16.376385    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c"]
Oct 31 02:00:16.594421 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:16.594390    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd"]
Oct 31 02:00:16.793661 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:16.793610    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c" event={"ID":"822c5afc-7c59-422d-a24d-d1b682127173","Type":"ContainerStarted","Data":"e78e05002c6d23ce58d1cfb9c3e9757eb37eb4d09fa06648ad74b19993b31276"}
Oct 31 02:00:16.793842 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:16.793680    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c" event={"ID":"822c5afc-7c59-422d-a24d-d1b682127173","Type":"ContainerStarted","Data":"040058f7441110dfac0c2c0ab9d91346e4f592cdcdafff125ddca6192e0f7883"}
Oct 31 02:00:16.793842 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:16.793702    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c"
Oct 31 02:00:16.795964 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:16.795933    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2" event={"ID":"e28001ed-ca1a-4790-96f5-73ada860e6de","Type":"ContainerStarted","Data":"f4e181d05f19647b7012b6982c0f2236f2a379fb0fdb02e85819465533b26ce1"}
Oct 31 02:00:16.797864 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:16.797833    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd" event={"ID":"f6eae8ee-0ae3-4beb-9777-e63e8853a9cb","Type":"ContainerStarted","Data":"c56ab6085f2b62b923413d5d1d4c09ead0d01ea13ce894601c817df50cf82034"}
Oct 31 02:00:16.798024 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:16.797885    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd" event={"ID":"f6eae8ee-0ae3-4beb-9777-e63e8853a9cb","Type":"ContainerStarted","Data":"bf8b8a595598a05f222b42632b2fc4d42a7deb0099871241eafe5fa3d055b709"}
Oct 31 02:00:16.798024 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:16.797913    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd"
Oct 31 02:00:16.829472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:16.824621    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c" podStartSLOduration=1.8246005730000001 podStartE2EDuration="1.824600573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:16.823767912 +0000 UTC m=+2372.827482203" watchObservedRunningTime="2026-10-31 02:00:16.824600573 +0000 UTC m=+2372.828314865"
Oct 31 02:00:16.867612 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:16.867551    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd" podStartSLOduration=1.867535309 podStartE2EDuration="1.867535309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:16.863977058 +0000 UTC m=+2372.867691349" watchObservedRunningTime="2026-10-31 02:00:16.867535309 +0000 UTC m=+2372.871249656"
Oct 31 02:00:17.130092 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:17.130001    5650 scope.go:122] "RemoveContainer" containerID="c456e81dd3a67bbe10e6610957a6816922da32a9aa56642a9b4d6dc78e3fbf84"
Oct 31 02:00:17.131201 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:17.130970    5650 scope.go:122] "RemoveContainer" containerID="3777de98872d2de1506658778cbe8dfd096767e00d3647e08b6b1c4328e80520"
Oct 31 02:00:17.132122 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:17.132102    5650 scope.go:122] "RemoveContainer" containerID="afe0af3bef6499d0363dccc5eb7e55706522540a17ac993c650e6d3d220c7a75"
Oct 31 02:00:17.134551 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:17.134529    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2"]
Oct 31 02:00:17.835209 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:17.835178    5650 generic.go:356] "Generic (PLEG): container finished" podID="e28001ed-ca1a-4790-96f5-73ada860e6de" containerID="f4e181d05f19647b7012b6982c0f2236f2a379fb0fdb02e85819465533b26ce1" exitCode=60
Oct 31 02:00:17.835731 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:17.835260    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2" event={"ID":"e28001ed-ca1a-4790-96f5-73ada860e6de","Type":"ContainerDied","Data":"f4e181d05f19647b7012b6982c0f2236f2a379fb0fdb02e85819465533b26ce1"}
Oct 31 02:00:17.835731 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:17.835323    5650 scope.go:122] "RemoveContainer" containerID="398ff41d6ac5fab8e3c6e0ffb7136a036a53b98a590d9f1464b4216faee951f5"
Oct 31 02:00:18.080697 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:18.080667    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2"
Oct 31 02:00:18.141528 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:18.133105    5650 scope.go:122] "RemoveContainer" containerID="e8df67abaa157b23d09be95d88cf22855209316ab7d8b4f8983b9b998b8ad35a"
Oct 31 02:00:18.177679 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:18.176934    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e28001ed-ca1a-4790-96f5-73ada860e6de-kube-api-access-tgnmd\" (UniqueName: \"kubernetes.io/projected/e28001ed-ca1a-4790-96f5-73ada860e6de-kube-api-access-tgnmd\") pod \"e28001ed-ca1a-4790-96f5-73ada860e6de\" (UID: \"e28001ed-ca1a-4790-96f5-73ada860e6de\") "
Oct 31 02:00:18.177679 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:18.177101    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e28001ed-ca1a-4790-96f5-73ada860e6de-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e28001ed-ca1a-4790-96f5-73ada860e6de-maas-api-service-ca\") pod \"e28001ed-ca1a-4790-96f5-73ada860e6de\" (UID: \"e28001ed-ca1a-4790-96f5-73ada860e6de\") "
Oct 31 02:00:18.177679 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:18.177603    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e28001ed-ca1a-4790-96f5-73ada860e6de-maas-api-service-ca" pod "e28001ed-ca1a-4790-96f5-73ada860e6de" (UID: "e28001ed-ca1a-4790-96f5-73ada860e6de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:18.178308 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:18.178275    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e28001ed-ca1a-4790-96f5-73ada860e6de-maas-api-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:18.185633 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:18.185562    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28001ed-ca1a-4790-96f5-73ada860e6de-kube-api-access-tgnmd" pod "e28001ed-ca1a-4790-96f5-73ada860e6de" (UID: "e28001ed-ca1a-4790-96f5-73ada860e6de"). InnerVolumeSpecName "kube-api-access-tgnmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:18.280938 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:18.280894    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgnmd\" (UniqueName: \"kubernetes.io/projected/e28001ed-ca1a-4790-96f5-73ada860e6de-kube-api-access-tgnmd\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:18.847629 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:18.847586    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw" event={"ID":"c5c76130-63ba-44f3-aa9e-c4ee0bceccec","Type":"ContainerStarted","Data":"a860a5e5d6f2c5dd1f416fed31fdbdaa02c9f6855c0dd2a9fd6eccbcc2dcb55b"}
Oct 31 02:00:18.849803 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:18.849773    5650 generic.go:356] "Generic (PLEG): container finished" podID="c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d" containerID="bb4cfb373978cb8a543b3555dd085d0768a2c266e910375441e70e1c38a73a08" exitCode=60
Oct 31 02:00:18.849933 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:18.849840    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49" event={"ID":"c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d","Type":"ContainerDied","Data":"bb4cfb373978cb8a543b3555dd085d0768a2c266e910375441e70e1c38a73a08"}
Oct 31 02:00:18.849933 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:18.849896    5650 scope.go:122] "RemoveContainer" containerID="c456e81dd3a67bbe10e6610957a6816922da32a9aa56642a9b4d6dc78e3fbf84"
Oct 31 02:00:18.850429 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:18.850324    5650 scope.go:122] "RemoveContainer" containerID="bb4cfb373978cb8a543b3555dd085d0768a2c266e910375441e70e1c38a73a08"
Oct 31 02:00:18.850850 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:00:18.850825    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49_odh-ai-gateway-infra(c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49" podUID="c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d"
Oct 31 02:00:18.854325 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:18.853719    5650 generic.go:356] "Generic (PLEG): container finished" podID="d87648fb-f2fb-42b8-b36b-22381fac5d7d" containerID="0302fc9e7f48936aa4f6c81aa8fbea55e111e4135d275f9c15a6bb96ccc8a2e9" exitCode=60
Oct 31 02:00:18.854325 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:18.853852    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg" event={"ID":"d87648fb-f2fb-42b8-b36b-22381fac5d7d","Type":"ContainerDied","Data":"0302fc9e7f48936aa4f6c81aa8fbea55e111e4135d275f9c15a6bb96ccc8a2e9"}
Oct 31 02:00:18.867243 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:18.862440    5650 scope.go:122] "RemoveContainer" containerID="0302fc9e7f48936aa4f6c81aa8fbea55e111e4135d275f9c15a6bb96ccc8a2e9"
Oct 31 02:00:18.867243 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:00:18.863090    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg_odh-ai-gateway-infra(d87648fb-f2fb-42b8-b36b-22381fac5d7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg" podUID="d87648fb-f2fb-42b8-b36b-22381fac5d7d"
Oct 31 02:00:18.918830 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:18.918794    5650 generic.go:356] "Generic (PLEG): container finished" podID="b9080245-f30d-4cb3-8c3a-61ea60ae2dd1" containerID="26fbc579ecdcdb9505a503372871c479bc842d6e6efc1574be29be2f2596d72e" exitCode=60
Oct 31 02:00:18.918996 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:18.918935    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw" event={"ID":"b9080245-f30d-4cb3-8c3a-61ea60ae2dd1","Type":"ContainerDied","Data":"26fbc579ecdcdb9505a503372871c479bc842d6e6efc1574be29be2f2596d72e"}
Oct 31 02:00:18.919669 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:18.919600    5650 scope.go:122] "RemoveContainer" containerID="26fbc579ecdcdb9505a503372871c479bc842d6e6efc1574be29be2f2596d72e"
Oct 31 02:00:18.920162 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:00:18.920074    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw_odh-ai-gateway-infra(b9080245-f30d-4cb3-8c3a-61ea60ae2dd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw" podUID="b9080245-f30d-4cb3-8c3a-61ea60ae2dd1"
Oct 31 02:00:18.923704 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:18.923682    5650 scope.go:122] "RemoveContainer" containerID="3777de98872d2de1506658778cbe8dfd096767e00d3647e08b6b1c4328e80520"
Oct 31 02:00:18.928825 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:18.928796    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2" event={"ID":"e28001ed-ca1a-4790-96f5-73ada860e6de","Type":"ContainerDied","Data":"bf51f391ba7b66c29d95c62155255765ef64c0f02e14bbc84da2f1847347dbbd"}
Oct 31 02:00:18.928981 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:18.928903    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2"
Oct 31 02:00:18.952819 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:18.952786    5650 scope.go:122] "RemoveContainer" containerID="afe0af3bef6499d0363dccc5eb7e55706522540a17ac993c650e6d3d220c7a75"
Oct 31 02:00:18.976020 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:18.975992    5650 scope.go:122] "RemoveContainer" containerID="f4e181d05f19647b7012b6982c0f2236f2a379fb0fdb02e85819465533b26ce1"
Oct 31 02:00:18.996394 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:18.996340    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2"]
Oct 31 02:00:19.003964 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:19.003927    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0b6abf-29890200-r9wx2"]
Oct 31 02:00:19.110998 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:19.110966    5650 scope.go:122] "RemoveContainer" containerID="ba8301509b47f9a815523adbc6e2df712c296f8c9ee67dc06a0945fc8d8233dc"
Oct 31 02:00:19.111174 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:19.111122    5650 scope.go:122] "RemoveContainer" containerID="1229c47c87eb61bf0851bb7a17eca95625526272d0b5435e31ea8d16cb719d6e"
Oct 31 02:00:19.111318 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:19.111296    5650 scope.go:122] "RemoveContainer" containerID="c3d2d3dcc6e42b4884f39939ff10101b04693995ce4ebd06951e68a20474b4ec"
Oct 31 02:00:19.111487 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:19.111467    5650 scope.go:122] "RemoveContainer" containerID="bc833d52334c40098a6aa4bfb295a84ace899d27cf5568bb7801d7848119684c"
Oct 31 02:00:19.896782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:19.896672    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw"]
Oct 31 02:00:19.930343 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:19.930182    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg"]
Oct 31 02:00:19.945398 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:19.945310    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6" event={"ID":"8f86643f-3419-470f-a3e9-4aae61fc266c","Type":"ContainerStarted","Data":"5caebb433e063f4229f2f0091a0bd00beb16ca12aff3daf9b2205c79dd868fe9"}
Oct 31 02:00:19.947884 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:19.947849    5650 generic.go:356] "Generic (PLEG): container finished" podID="c5c76130-63ba-44f3-aa9e-c4ee0bceccec" containerID="a860a5e5d6f2c5dd1f416fed31fdbdaa02c9f6855c0dd2a9fd6eccbcc2dcb55b" exitCode=60
Oct 31 02:00:19.948021 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:19.947882    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw" event={"ID":"c5c76130-63ba-44f3-aa9e-c4ee0bceccec","Type":"ContainerDied","Data":"a860a5e5d6f2c5dd1f416fed31fdbdaa02c9f6855c0dd2a9fd6eccbcc2dcb55b"}
Oct 31 02:00:19.948021 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:19.947955    5650 scope.go:122] "RemoveContainer" containerID="e8df67abaa157b23d09be95d88cf22855209316ab7d8b4f8983b9b998b8ad35a"
Oct 31 02:00:19.959714 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:19.959676    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh" event={"ID":"2bcc823c-fc5f-47c0-9cfa-665f05c71056","Type":"ContainerStarted","Data":"c79b9532ece2759827ebd2dc393b90688af64a2c838e7147db343ed4311218bd"}
Oct 31 02:00:19.961940 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:19.961910    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49"]
Oct 31 02:00:19.973333 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:19.973296    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf" event={"ID":"b428ad4f-cd43-4daa-8f01-e94c7ef63732","Type":"ContainerStarted","Data":"ef4644378473429e46cae4ff0dea2974475df2e51fde993be1f57db0df6f6173"}
Oct 31 02:00:20.013483 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.012866    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw"]
Oct 31 02:00:20.164040 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.163940    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e28001ed-ca1a-4790-96f5-73ada860e6de" path="/var/lib/kubelet/pods/e28001ed-ca1a-4790-96f5-73ada860e6de/volumes"
Oct 31 02:00:20.297144 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.295307    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw"
Oct 31 02:00:20.357308 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.357251    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5c76130-63ba-44f3-aa9e-c4ee0bceccec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5c76130-63ba-44f3-aa9e-c4ee0bceccec-maas-api-service-ca\") pod \"c5c76130-63ba-44f3-aa9e-c4ee0bceccec\" (UID: \"c5c76130-63ba-44f3-aa9e-c4ee0bceccec\") "
Oct 31 02:00:20.357776 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.357751    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c76130-63ba-44f3-aa9e-c4ee0bceccec-kube-api-access-j28xx\" (UniqueName: \"kubernetes.io/projected/c5c76130-63ba-44f3-aa9e-c4ee0bceccec-kube-api-access-j28xx\") pod \"c5c76130-63ba-44f3-aa9e-c4ee0bceccec\" (UID: \"c5c76130-63ba-44f3-aa9e-c4ee0bceccec\") "
Oct 31 02:00:20.359151 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.359119    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c76130-63ba-44f3-aa9e-c4ee0bceccec-maas-api-service-ca" pod "c5c76130-63ba-44f3-aa9e-c4ee0bceccec" (UID: "c5c76130-63ba-44f3-aa9e-c4ee0bceccec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:20.360705 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.360676    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5c76130-63ba-44f3-aa9e-c4ee0bceccec-maas-api-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:20.380769 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.378606    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c76130-63ba-44f3-aa9e-c4ee0bceccec-kube-api-access-j28xx" pod "c5c76130-63ba-44f3-aa9e-c4ee0bceccec" (UID: "c5c76130-63ba-44f3-aa9e-c4ee0bceccec"). InnerVolumeSpecName "kube-api-access-j28xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:20.463485 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.463410    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j28xx\" (UniqueName: \"kubernetes.io/projected/c5c76130-63ba-44f3-aa9e-c4ee0bceccec-kube-api-access-j28xx\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:20.480753 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.480723    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49"
Oct 31 02:00:20.484631 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.484599    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw"
Oct 31 02:00:20.486638 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.486615    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg"
Oct 31 02:00:20.684503 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.684340    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d-maas-api-service-ca\") pod \"c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d\" (UID: \"c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d\") "
Oct 31 02:00:20.684503 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.684464    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9080245-f30d-4cb3-8c3a-61ea60ae2dd1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9080245-f30d-4cb3-8c3a-61ea60ae2dd1-maas-api-service-ca\") pod \"b9080245-f30d-4cb3-8c3a-61ea60ae2dd1\" (UID: \"b9080245-f30d-4cb3-8c3a-61ea60ae2dd1\") "
Oct 31 02:00:20.684743 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.684510    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9080245-f30d-4cb3-8c3a-61ea60ae2dd1-kube-api-access-ft2bk\" (UniqueName: \"kubernetes.io/projected/b9080245-f30d-4cb3-8c3a-61ea60ae2dd1-kube-api-access-ft2bk\") pod \"b9080245-f30d-4cb3-8c3a-61ea60ae2dd1\" (UID: \"b9080245-f30d-4cb3-8c3a-61ea60ae2dd1\") "
Oct 31 02:00:20.684743 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.684607    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d87648fb-f2fb-42b8-b36b-22381fac5d7d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87648fb-f2fb-42b8-b36b-22381fac5d7d-maas-api-service-ca\") pod \"d87648fb-f2fb-42b8-b36b-22381fac5d7d\" (UID: \"d87648fb-f2fb-42b8-b36b-22381fac5d7d\") "
Oct 31 02:00:20.684743 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.684710    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d87648fb-f2fb-42b8-b36b-22381fac5d7d-kube-api-access-hb6mr\" (UniqueName: \"kubernetes.io/projected/d87648fb-f2fb-42b8-b36b-22381fac5d7d-kube-api-access-hb6mr\") pod \"d87648fb-f2fb-42b8-b36b-22381fac5d7d\" (UID: \"d87648fb-f2fb-42b8-b36b-22381fac5d7d\") "
Oct 31 02:00:20.684932 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.684914    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d-kube-api-access-cfpk9\" (UniqueName: \"kubernetes.io/projected/c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d-kube-api-access-cfpk9\") pod \"c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d\" (UID: \"c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d\") "
Oct 31 02:00:20.685340 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.685305    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9080245-f30d-4cb3-8c3a-61ea60ae2dd1-maas-api-service-ca" pod "b9080245-f30d-4cb3-8c3a-61ea60ae2dd1" (UID: "b9080245-f30d-4cb3-8c3a-61ea60ae2dd1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:20.686524 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.686376    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9080245-f30d-4cb3-8c3a-61ea60ae2dd1-maas-api-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:20.686950 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.686922    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d-maas-api-service-ca" pod "c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d" (UID: "c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:20.687281 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.687252    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d87648fb-f2fb-42b8-b36b-22381fac5d7d-maas-api-service-ca" pod "d87648fb-f2fb-42b8-b36b-22381fac5d7d" (UID: "d87648fb-f2fb-42b8-b36b-22381fac5d7d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:20.688252 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.688223    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d-kube-api-access-cfpk9" pod "c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d" (UID: "c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d"). InnerVolumeSpecName "kube-api-access-cfpk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:20.688885 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.688859    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d87648fb-f2fb-42b8-b36b-22381fac5d7d-kube-api-access-hb6mr" pod "d87648fb-f2fb-42b8-b36b-22381fac5d7d" (UID: "d87648fb-f2fb-42b8-b36b-22381fac5d7d"). InnerVolumeSpecName "kube-api-access-hb6mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:20.689295 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.689264    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9080245-f30d-4cb3-8c3a-61ea60ae2dd1-kube-api-access-ft2bk" pod "b9080245-f30d-4cb3-8c3a-61ea60ae2dd1" (UID: "b9080245-f30d-4cb3-8c3a-61ea60ae2dd1"). InnerVolumeSpecName "kube-api-access-ft2bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:20.788231 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.788184    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87648fb-f2fb-42b8-b36b-22381fac5d7d-maas-api-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:20.788402 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.788261    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb6mr\" (UniqueName: \"kubernetes.io/projected/d87648fb-f2fb-42b8-b36b-22381fac5d7d-kube-api-access-hb6mr\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:20.788402 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.788299    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfpk9\" (UniqueName: \"kubernetes.io/projected/c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d-kube-api-access-cfpk9\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:20.788402 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.788308    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d-maas-api-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:20.788402 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.788318    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft2bk\" (UniqueName: \"kubernetes.io/projected/b9080245-f30d-4cb3-8c3a-61ea60ae2dd1-kube-api-access-ft2bk\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:20.984946 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.984915    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6"]
Oct 31 02:00:20.986128 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.986106    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49"
Oct 31 02:00:20.986253 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.986120    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49" event={"ID":"c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d","Type":"ContainerDied","Data":"9885d03b54ea71ba1b86fdd5157468b46b5c18d8a5fc91907c5536be8be2b308"}
Oct 31 02:00:20.986253 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.986183    5650 scope.go:122] "RemoveContainer" containerID="bb4cfb373978cb8a543b3555dd085d0768a2c266e910375441e70e1c38a73a08"
Oct 31 02:00:20.988602 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.988578    5650 generic.go:356] "Generic (PLEG): container finished" podID="2bcc823c-fc5f-47c0-9cfa-665f05c71056" containerID="c79b9532ece2759827ebd2dc393b90688af64a2c838e7147db343ed4311218bd" exitCode=60
Oct 31 02:00:20.988727 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.988651    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh" event={"ID":"2bcc823c-fc5f-47c0-9cfa-665f05c71056","Type":"ContainerDied","Data":"c79b9532ece2759827ebd2dc393b90688af64a2c838e7147db343ed4311218bd"}
Oct 31 02:00:20.989306 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.989282    5650 scope.go:122] "RemoveContainer" containerID="c79b9532ece2759827ebd2dc393b90688af64a2c838e7147db343ed4311218bd"
Oct 31 02:00:20.989843 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:00:20.989790    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh_odh-ai-gateway-infra(2bcc823c-fc5f-47c0-9cfa-665f05c71056)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh" podUID="2bcc823c-fc5f-47c0-9cfa-665f05c71056"
Oct 31 02:00:20.991084 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.991063    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg"
Oct 31 02:00:20.991299 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.991250    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg" event={"ID":"d87648fb-f2fb-42b8-b36b-22381fac5d7d","Type":"ContainerDied","Data":"05ebfa2770f11f3d80a4bbf452a745d3cde7371856939881acfd77950f887851"}
Oct 31 02:00:20.993014 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.992977    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw"
Oct 31 02:00:20.993596 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.993033    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw" event={"ID":"b9080245-f30d-4cb3-8c3a-61ea60ae2dd1","Type":"ContainerDied","Data":"df3a39723c252489c1327addef62d95c92afab0435e3b2087cb14b13755a9cb6"}
Oct 31 02:00:20.997794 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.997766    5650 generic.go:356] "Generic (PLEG): container finished" podID="b428ad4f-cd43-4daa-8f01-e94c7ef63732" containerID="ef4644378473429e46cae4ff0dea2974475df2e51fde993be1f57db0df6f6173" exitCode=60
Oct 31 02:00:20.997922 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.997817    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf" event={"ID":"b428ad4f-cd43-4daa-8f01-e94c7ef63732","Type":"ContainerDied","Data":"ef4644378473429e46cae4ff0dea2974475df2e51fde993be1f57db0df6f6173"}
Oct 31 02:00:20.998768 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:20.998336    5650 scope.go:122] "RemoveContainer" containerID="ef4644378473429e46cae4ff0dea2974475df2e51fde993be1f57db0df6f6173"
Oct 31 02:00:20.998874 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:00:20.998837    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf_odh-ai-gateway-infra(b428ad4f-cd43-4daa-8f01-e94c7ef63732)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf" podUID="b428ad4f-cd43-4daa-8f01-e94c7ef63732"
Oct 31 02:00:21.002750 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.002720    5650 generic.go:356] "Generic (PLEG): container finished" podID="8f86643f-3419-470f-a3e9-4aae61fc266c" containerID="5caebb433e063f4229f2f0091a0bd00beb16ca12aff3daf9b2205c79dd868fe9" exitCode=60
Oct 31 02:00:21.002875 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.002837    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6" event={"ID":"8f86643f-3419-470f-a3e9-4aae61fc266c","Type":"ContainerDied","Data":"5caebb433e063f4229f2f0091a0bd00beb16ca12aff3daf9b2205c79dd868fe9"}
Oct 31 02:00:21.004597 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.004577    5650 scope.go:122] "RemoveContainer" containerID="c3d2d3dcc6e42b4884f39939ff10101b04693995ce4ebd06951e68a20474b4ec"
Oct 31 02:00:21.008596 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.008566    5650 generic.go:356] "Generic (PLEG): container finished" podID="0c56489d-d1a0-4ae3-9260-eb0d18201d12" containerID="1960e2140d19017c49ff4c0a2a581a4c5e4795df8dfe6ff8d36881a0f8b0ade7" exitCode=60
Oct 31 02:00:21.008713 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.008654    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp" event={"ID":"0c56489d-d1a0-4ae3-9260-eb0d18201d12","Type":"ContainerDied","Data":"1960e2140d19017c49ff4c0a2a581a4c5e4795df8dfe6ff8d36881a0f8b0ade7"}
Oct 31 02:00:21.009249 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.009228    5650 scope.go:122] "RemoveContainer" containerID="1960e2140d19017c49ff4c0a2a581a4c5e4795df8dfe6ff8d36881a0f8b0ade7"
Oct 31 02:00:21.009796 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:00:21.009770    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp_odh-ai-gateway-infra(0c56489d-d1a0-4ae3-9260-eb0d18201d12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp" podUID="0c56489d-d1a0-4ae3-9260-eb0d18201d12"
Oct 31 02:00:21.012740 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.012632    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw"
Oct 31 02:00:21.012740 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.012715    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw" event={"ID":"c5c76130-63ba-44f3-aa9e-c4ee0bceccec","Type":"ContainerDied","Data":"4eead39c8bb3f651e1e90e37c6da3fd8739eaa8dbe757a543980ea7e172c50d6"}
Oct 31 02:00:21.025681 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.025644    5650 scope.go:122] "RemoveContainer" containerID="0302fc9e7f48936aa4f6c81aa8fbea55e111e4135d275f9c15a6bb96ccc8a2e9"
Oct 31 02:00:21.046336 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.046309    5650 scope.go:122] "RemoveContainer" containerID="26fbc579ecdcdb9505a503372871c479bc842d6e6efc1574be29be2f2596d72e"
Oct 31 02:00:21.063730 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.063617    5650 scope.go:122] "RemoveContainer" containerID="ba8301509b47f9a815523adbc6e2df712c296f8c9ee67dc06a0945fc8d8233dc"
Oct 31 02:00:21.085661 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.085632    5650 scope.go:122] "RemoveContainer" containerID="1229c47c87eb61bf0851bb7a17eca95625526272d0b5435e31ea8d16cb719d6e"
Oct 31 02:00:21.109485 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.109435    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg"]
Oct 31 02:00:21.112071 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.112049    5650 scope.go:122] "RemoveContainer" containerID="bc833d52334c40098a6aa4bfb295a84ace899d27cf5568bb7801d7848119684c"
Oct 31 02:00:21.114053 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.114021    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-be6935-29890200-d92fg"]
Oct 31 02:00:21.128466 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.128352    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh"]
Oct 31 02:00:21.136274 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.136242    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw"]
Oct 31 02:00:21.137490 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.137102    5650 scope.go:122] "RemoveContainer" containerID="a860a5e5d6f2c5dd1f416fed31fdbdaa02c9f6855c0dd2a9fd6eccbcc2dcb55b"
Oct 31 02:00:21.143720 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.143693    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890200-f6jxw"]
Oct 31 02:00:21.163709 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.163356    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw"]
Oct 31 02:00:21.171935 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.171902    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-89e955-29890200-jznhw"]
Oct 31 02:00:21.187401 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.187370    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49"]
Oct 31 02:00:21.229725 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.229615    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890200-9ht49"]
Oct 31 02:00:21.232417 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.232390    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6"
Oct 31 02:00:21.267051 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.266963    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf"]
Oct 31 02:00:21.327739 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.327695    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f86643f-3419-470f-a3e9-4aae61fc266c-kube-api-access-n8xlv\" (UniqueName: \"kubernetes.io/projected/8f86643f-3419-470f-a3e9-4aae61fc266c-kube-api-access-n8xlv\") pod \"8f86643f-3419-470f-a3e9-4aae61fc266c\" (UID: \"8f86643f-3419-470f-a3e9-4aae61fc266c\") "
Oct 31 02:00:21.327962 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.327808    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f86643f-3419-470f-a3e9-4aae61fc266c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f86643f-3419-470f-a3e9-4aae61fc266c-maas-api-service-ca\") pod \"8f86643f-3419-470f-a3e9-4aae61fc266c\" (UID: \"8f86643f-3419-470f-a3e9-4aae61fc266c\") "
Oct 31 02:00:21.328308 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.328277    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f86643f-3419-470f-a3e9-4aae61fc266c-maas-api-service-ca" pod "8f86643f-3419-470f-a3e9-4aae61fc266c" (UID: "8f86643f-3419-470f-a3e9-4aae61fc266c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:21.328783 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.328757    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f86643f-3419-470f-a3e9-4aae61fc266c-maas-api-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:21.330238 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.330212    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f86643f-3419-470f-a3e9-4aae61fc266c-kube-api-access-n8xlv" pod "8f86643f-3419-470f-a3e9-4aae61fc266c" (UID: "8f86643f-3419-470f-a3e9-4aae61fc266c"). InnerVolumeSpecName "kube-api-access-n8xlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:21.390615 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.390577    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 02:00:21.390803 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.390637    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww" podUID="1da683f3-ae35-4c60-a119-7711ca882f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:00:21.431541 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.431502    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8xlv\" (UniqueName: \"kubernetes.io/projected/8f86643f-3419-470f-a3e9-4aae61fc266c-kube-api-access-n8xlv\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:21.707548 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.707499    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"]
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.708962    5650 cpu_manager.go:414] "containerMap: removing container" podUID="e28001ed-ca1a-4790-96f5-73ada860e6de" containerName="cleanup"
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.708990    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e28001ed-ca1a-4790-96f5-73ada860e6de" containerName="cleanup"
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.709015    5650 cpu_manager.go:414] "containerMap: removing container" podUID="c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d" containerName="cleanup"
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.709022    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d" containerName="cleanup"
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.709042    5650 cpu_manager.go:414] "containerMap: removing container" podUID="e28001ed-ca1a-4790-96f5-73ada860e6de" containerName="cleanup"
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.709049    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e28001ed-ca1a-4790-96f5-73ada860e6de" containerName="cleanup"
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.709066    5650 cpu_manager.go:414] "containerMap: removing container" podUID="8f86643f-3419-470f-a3e9-4aae61fc266c" containerName="cleanup"
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.709073    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f86643f-3419-470f-a3e9-4aae61fc266c" containerName="cleanup"
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.709090    5650 cpu_manager.go:414] "containerMap: removing container" podUID="8f86643f-3419-470f-a3e9-4aae61fc266c" containerName="cleanup"
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.709096    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f86643f-3419-470f-a3e9-4aae61fc266c" containerName="cleanup"
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.709174    5650 cpu_manager.go:414] "containerMap: removing container" podUID="b9080245-f30d-4cb3-8c3a-61ea60ae2dd1" containerName="cleanup"
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.709184    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9080245-f30d-4cb3-8c3a-61ea60ae2dd1" containerName="cleanup"
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.709194    5650 cpu_manager.go:414] "containerMap: removing container" podUID="8f86643f-3419-470f-a3e9-4aae61fc266c" containerName="cleanup"
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.709201    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f86643f-3419-470f-a3e9-4aae61fc266c" containerName="cleanup"
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.709225    5650 cpu_manager.go:414] "containerMap: removing container" podUID="d87648fb-f2fb-42b8-b36b-22381fac5d7d" containerName="cleanup"
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.709231    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d87648fb-f2fb-42b8-b36b-22381fac5d7d" containerName="cleanup"
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.709249    5650 cpu_manager.go:414] "containerMap: removing container" podUID="b9080245-f30d-4cb3-8c3a-61ea60ae2dd1" containerName="cleanup"
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.709256    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9080245-f30d-4cb3-8c3a-61ea60ae2dd1" containerName="cleanup"
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.709270    5650 cpu_manager.go:414] "containerMap: removing container" podUID="c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d" containerName="cleanup"
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.709276    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d" containerName="cleanup"
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.709291    5650 cpu_manager.go:414] "containerMap: removing container" podUID="b9080245-f30d-4cb3-8c3a-61ea60ae2dd1" containerName="cleanup"
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.709297    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9080245-f30d-4cb3-8c3a-61ea60ae2dd1" containerName="cleanup"
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.709308    5650 cpu_manager.go:414] "containerMap: removing container" podUID="c5c76130-63ba-44f3-aa9e-c4ee0bceccec" containerName="cleanup"
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.709315    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c76130-63ba-44f3-aa9e-c4ee0bceccec" containerName="cleanup"
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.709331    5650 cpu_manager.go:414] "containerMap: removing container" podUID="c5c76130-63ba-44f3-aa9e-c4ee0bceccec" containerName="cleanup"
Oct 31 02:00:21.709901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.709338    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c76130-63ba-44f3-aa9e-c4ee0bceccec" containerName="cleanup"
Oct 31 02:00:21.717782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.711571    5650 cpu_manager.go:414] "containerMap: removing container" podUID="c5c76130-63ba-44f3-aa9e-c4ee0bceccec" containerName="cleanup"
Oct 31 02:00:21.717782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.711599    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c76130-63ba-44f3-aa9e-c4ee0bceccec" containerName="cleanup"
Oct 31 02:00:21.717782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.711632    5650 cpu_manager.go:414] "containerMap: removing container" podUID="d87648fb-f2fb-42b8-b36b-22381fac5d7d" containerName="cleanup"
Oct 31 02:00:21.717782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.711641    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d87648fb-f2fb-42b8-b36b-22381fac5d7d" containerName="cleanup"
Oct 31 02:00:21.717782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.711916    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="e28001ed-ca1a-4790-96f5-73ada860e6de" containerName="cleanup"
Oct 31 02:00:21.717782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.711935    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d" containerName="cleanup"
Oct 31 02:00:21.717782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.711950    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="e28001ed-ca1a-4790-96f5-73ada860e6de" containerName="cleanup"
Oct 31 02:00:21.717782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.711960    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f86643f-3419-470f-a3e9-4aae61fc266c" containerName="cleanup"
Oct 31 02:00:21.717782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.711971    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f86643f-3419-470f-a3e9-4aae61fc266c" containerName="cleanup"
Oct 31 02:00:21.717782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.711987    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9080245-f30d-4cb3-8c3a-61ea60ae2dd1" containerName="cleanup"
Oct 31 02:00:21.717782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.712001    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="d87648fb-f2fb-42b8-b36b-22381fac5d7d" containerName="cleanup"
Oct 31 02:00:21.717782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.712013    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9080245-f30d-4cb3-8c3a-61ea60ae2dd1" containerName="cleanup"
Oct 31 02:00:21.717782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.712021    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="e28001ed-ca1a-4790-96f5-73ada860e6de" containerName="cleanup"
Oct 31 02:00:21.717782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.712035    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="d87648fb-f2fb-42b8-b36b-22381fac5d7d" containerName="cleanup"
Oct 31 02:00:21.717782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.712048    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9080245-f30d-4cb3-8c3a-61ea60ae2dd1" containerName="cleanup"
Oct 31 02:00:21.717782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.712055    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c76130-63ba-44f3-aa9e-c4ee0bceccec" containerName="cleanup"
Oct 31 02:00:21.717782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.712065    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c76130-63ba-44f3-aa9e-c4ee0bceccec" containerName="cleanup"
Oct 31 02:00:21.717782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.712076    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c76130-63ba-44f3-aa9e-c4ee0bceccec" containerName="cleanup"
Oct 31 02:00:21.717782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.712086    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="d87648fb-f2fb-42b8-b36b-22381fac5d7d" containerName="cleanup"
Oct 31 02:00:21.717782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.712100    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d" containerName="cleanup"
Oct 31 02:00:21.727397 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.727358    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.729673 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.729234    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"]
Oct 31 02:00:21.738985 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.738944    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-bb5c51-openshift-default-dockercfg-s2hdn\""
Oct 31 02:00:21.840468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.839825    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.840468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.839887    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.840468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.839938    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.840468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.840046    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.840468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.840095    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.840468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.840139    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.840468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.840219    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b62lh\" (UniqueName: \"kubernetes.io/projected/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-kube-api-access-b62lh\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.840468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.840289    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.840468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.840385    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.945467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.943731    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.945467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.943795    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.945467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.943839    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.945467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.943934    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.945467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.943978    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.945467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.944019    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.945467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.944100    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b62lh\" (UniqueName: \"kubernetes.io/projected/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-kube-api-access-b62lh\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.945467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.944167    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.945467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.944261    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.945467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.945266    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.946079 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.945547    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.949467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.946183    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.949467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.946791    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.949467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.947053    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.978072 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.977982    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.991814 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.989062    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.993095 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.993064    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b62lh\" (UniqueName: \"kubernetes.io/projected/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-kube-api-access-b62lh\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:21.993557 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:21.993531    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:22.051503 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.051295    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:22.074537 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.074492    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp"]
Oct 31 02:00:22.080165 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.080132    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6" event={"ID":"8f86643f-3419-470f-a3e9-4aae61fc266c","Type":"ContainerDied","Data":"74b96405b086ab6f5ef920d4931bc0e39c12402b4e65ca773587261c806dc7af"}
Oct 31 02:00:22.080533 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.080512    5650 scope.go:122] "RemoveContainer" containerID="5caebb433e063f4229f2f0091a0bd00beb16ca12aff3daf9b2205c79dd868fe9"
Oct 31 02:00:22.080787 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.080772    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6"
Oct 31 02:00:22.139040 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.139004    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9080245-f30d-4cb3-8c3a-61ea60ae2dd1" path="/var/lib/kubelet/pods/b9080245-f30d-4cb3-8c3a-61ea60ae2dd1/volumes"
Oct 31 02:00:22.139749 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.139725    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d" path="/var/lib/kubelet/pods/c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d/volumes"
Oct 31 02:00:22.140337 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.140316    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5c76130-63ba-44f3-aa9e-c4ee0bceccec" path="/var/lib/kubelet/pods/c5c76130-63ba-44f3-aa9e-c4ee0bceccec/volumes"
Oct 31 02:00:22.148869 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.145765    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d87648fb-f2fb-42b8-b36b-22381fac5d7d" path="/var/lib/kubelet/pods/d87648fb-f2fb-42b8-b36b-22381fac5d7d/volumes"
Oct 31 02:00:22.177335 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.177285    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6"]
Oct 31 02:00:22.188813 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.188775    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae5a1c-29890200-6dgc6"]
Oct 31 02:00:22.510173 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.510133    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp"
Oct 31 02:00:22.579355 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.579279    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf"
Oct 31 02:00:22.626572 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.625825    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c56489d-d1a0-4ae3-9260-eb0d18201d12-kube-api-access-lvmvh\" (UniqueName: \"kubernetes.io/projected/0c56489d-d1a0-4ae3-9260-eb0d18201d12-kube-api-access-lvmvh\") pod \"0c56489d-d1a0-4ae3-9260-eb0d18201d12\" (UID: \"0c56489d-d1a0-4ae3-9260-eb0d18201d12\") "
Oct 31 02:00:22.626572 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.626131    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c56489d-d1a0-4ae3-9260-eb0d18201d12-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c56489d-d1a0-4ae3-9260-eb0d18201d12-maas-api-service-ca\") pod \"0c56489d-d1a0-4ae3-9260-eb0d18201d12\" (UID: \"0c56489d-d1a0-4ae3-9260-eb0d18201d12\") "
Oct 31 02:00:22.629300 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.629249    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c56489d-d1a0-4ae3-9260-eb0d18201d12-maas-api-service-ca" pod "0c56489d-d1a0-4ae3-9260-eb0d18201d12" (UID: "0c56489d-d1a0-4ae3-9260-eb0d18201d12"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:22.638528 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.638458    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c56489d-d1a0-4ae3-9260-eb0d18201d12-kube-api-access-lvmvh" pod "0c56489d-d1a0-4ae3-9260-eb0d18201d12" (UID: "0c56489d-d1a0-4ae3-9260-eb0d18201d12"). InnerVolumeSpecName "kube-api-access-lvmvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:22.664788 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.664191    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh"
Oct 31 02:00:22.729315 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.729269    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b428ad4f-cd43-4daa-8f01-e94c7ef63732-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b428ad4f-cd43-4daa-8f01-e94c7ef63732-maas-api-service-ca\") pod \"b428ad4f-cd43-4daa-8f01-e94c7ef63732\" (UID: \"b428ad4f-cd43-4daa-8f01-e94c7ef63732\") "
Oct 31 02:00:22.729532 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.729431    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b428ad4f-cd43-4daa-8f01-e94c7ef63732-kube-api-access-j7s9z\" (UniqueName: \"kubernetes.io/projected/b428ad4f-cd43-4daa-8f01-e94c7ef63732-kube-api-access-j7s9z\") pod \"b428ad4f-cd43-4daa-8f01-e94c7ef63732\" (UID: \"b428ad4f-cd43-4daa-8f01-e94c7ef63732\") "
Oct 31 02:00:22.729741 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.729718    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b428ad4f-cd43-4daa-8f01-e94c7ef63732-maas-api-service-ca" pod "b428ad4f-cd43-4daa-8f01-e94c7ef63732" (UID: "b428ad4f-cd43-4daa-8f01-e94c7ef63732"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:22.730546 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.730512    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c56489d-d1a0-4ae3-9260-eb0d18201d12-maas-api-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:22.730664 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.730545    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvmvh\" (UniqueName: \"kubernetes.io/projected/0c56489d-d1a0-4ae3-9260-eb0d18201d12-kube-api-access-lvmvh\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:22.730664 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.730565    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b428ad4f-cd43-4daa-8f01-e94c7ef63732-maas-api-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:22.732138 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.732108    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b428ad4f-cd43-4daa-8f01-e94c7ef63732-kube-api-access-j7s9z" pod "b428ad4f-cd43-4daa-8f01-e94c7ef63732" (UID: "b428ad4f-cd43-4daa-8f01-e94c7ef63732"). InnerVolumeSpecName "kube-api-access-j7s9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:22.831894 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.831798    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bcc823c-fc5f-47c0-9cfa-665f05c71056-kube-api-access-75n9f\" (UniqueName: \"kubernetes.io/projected/2bcc823c-fc5f-47c0-9cfa-665f05c71056-kube-api-access-75n9f\") pod \"2bcc823c-fc5f-47c0-9cfa-665f05c71056\" (UID: \"2bcc823c-fc5f-47c0-9cfa-665f05c71056\") "
Oct 31 02:00:22.831894 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.831861    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bcc823c-fc5f-47c0-9cfa-665f05c71056-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bcc823c-fc5f-47c0-9cfa-665f05c71056-maas-api-service-ca\") pod \"2bcc823c-fc5f-47c0-9cfa-665f05c71056\" (UID: \"2bcc823c-fc5f-47c0-9cfa-665f05c71056\") "
Oct 31 02:00:22.832352 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.832321    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bcc823c-fc5f-47c0-9cfa-665f05c71056-maas-api-service-ca" pod "2bcc823c-fc5f-47c0-9cfa-665f05c71056" (UID: "2bcc823c-fc5f-47c0-9cfa-665f05c71056"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:22.832869 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.832843    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bcc823c-fc5f-47c0-9cfa-665f05c71056-maas-api-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:22.832869 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.832867    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7s9z\" (UniqueName: \"kubernetes.io/projected/b428ad4f-cd43-4daa-8f01-e94c7ef63732-kube-api-access-j7s9z\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:22.834243 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.834212    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bcc823c-fc5f-47c0-9cfa-665f05c71056-kube-api-access-75n9f" pod "2bcc823c-fc5f-47c0-9cfa-665f05c71056" (UID: "2bcc823c-fc5f-47c0-9cfa-665f05c71056"). InnerVolumeSpecName "kube-api-access-75n9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:22.879078 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:00:22.879030    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb82515_c5d2_47a6_af96_d0da5a75bcd3.slice/crio-d6f070103fbe1062288de64cb7fdd768aa5dd9978f96588c2bce9e8a6ca9c799 WatchSource:0}: Error finding container d6f070103fbe1062288de64cb7fdd768aa5dd9978f96588c2bce9e8a6ca9c799: Status 404 returned error can't find the container with id d6f070103fbe1062288de64cb7fdd768aa5dd9978f96588c2bce9e8a6ca9c799
Oct 31 02:00:22.881674 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.881632    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"]
Oct 31 02:00:22.883056 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.882511    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:00:22.883056 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.882594    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:00:22.883056 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.882632    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:00:22.935096 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:22.935060    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75n9f\" (UniqueName: \"kubernetes.io/projected/2bcc823c-fc5f-47c0-9cfa-665f05c71056-kube-api-access-75n9f\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:00:23.138699 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:23.135487    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf" event={"ID":"b428ad4f-cd43-4daa-8f01-e94c7ef63732","Type":"ContainerDied","Data":"b9ef2d4c310382cba83b758bc43044855841af22693a0b905ae08bc8772470f0"}
Oct 31 02:00:23.138699 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:23.135564    5650 scope.go:122] "RemoveContainer" containerID="ef4644378473429e46cae4ff0dea2974475df2e51fde993be1f57db0df6f6173"
Oct 31 02:00:23.138699 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:23.135675    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf"
Oct 31 02:00:23.143737 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:23.143705    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m" event={"ID":"2cb82515-c5d2-47a6-af96-d0da5a75bcd3","Type":"ContainerStarted","Data":"722b81cc31875b9ef514423be1993eaa0c5106f7b32341981e17336b1ec009f8"}
Oct 31 02:00:23.143891 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:23.143766    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m" event={"ID":"2cb82515-c5d2-47a6-af96-d0da5a75bcd3","Type":"ContainerStarted","Data":"d6f070103fbe1062288de64cb7fdd768aa5dd9978f96588c2bce9e8a6ca9c799"}
Oct 31 02:00:23.148304 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:23.146579    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp"
Oct 31 02:00:23.148304 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:23.146595    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp" event={"ID":"0c56489d-d1a0-4ae3-9260-eb0d18201d12","Type":"ContainerDied","Data":"41f05c973ccd4c743bad6cdb67fac18a809d7751f704987dbbbc98448416dbc8"}
Oct 31 02:00:23.150555 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:23.150415    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh" event={"ID":"2bcc823c-fc5f-47c0-9cfa-665f05c71056","Type":"ContainerDied","Data":"b9d3d3f06db8f2d7033b9b476ba4833b3f47c60252295c9fa62558dede767d59"}
Oct 31 02:00:23.150555 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:23.150501    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh"
Oct 31 02:00:23.154132 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:23.154099    5650 scope.go:122] "RemoveContainer" containerID="1960e2140d19017c49ff4c0a2a581a4c5e4795df8dfe6ff8d36881a0f8b0ade7"
Oct 31 02:00:23.167826 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:23.167494    5650 scope.go:122] "RemoveContainer" containerID="c79b9532ece2759827ebd2dc393b90688af64a2c838e7147db343ed4311218bd"
Oct 31 02:00:23.182476 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:23.182410    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m" podStartSLOduration=2.182395578 podStartE2EDuration="2.182395578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:23.176165408 +0000 UTC m=+2379.179879700" watchObservedRunningTime="2026-10-31 02:00:23.182395578 +0000 UTC m=+2379.186109869"
Oct 31 02:00:23.203929 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:23.201239    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp"]
Oct 31 02:00:23.231284 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:23.231245    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7e5955-29890200-hpvhp"]
Oct 31 02:00:23.241333 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:23.241292    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh"]
Oct 31 02:00:23.246223 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:23.246189    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-46a67e-29890200-894jh"]
Oct 31 02:00:23.260524 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:23.260496    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf"]
Oct 31 02:00:23.263952 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:23.263919    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ac7e0d-29890200-l4hcf"]
Oct 31 02:00:24.072600 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:24.069252    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:24.094116 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:24.094057    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:24.122612 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:24.122573    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c56489d-d1a0-4ae3-9260-eb0d18201d12" path="/var/lib/kubelet/pods/0c56489d-d1a0-4ae3-9260-eb0d18201d12/volumes"
Oct 31 02:00:24.123484 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:24.123417    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bcc823c-fc5f-47c0-9cfa-665f05c71056" path="/var/lib/kubelet/pods/2bcc823c-fc5f-47c0-9cfa-665f05c71056/volumes"
Oct 31 02:00:24.124195 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:24.124153    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f86643f-3419-470f-a3e9-4aae61fc266c" path="/var/lib/kubelet/pods/8f86643f-3419-470f-a3e9-4aae61fc266c/volumes"
Oct 31 02:00:24.125341 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:24.125309    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b428ad4f-cd43-4daa-8f01-e94c7ef63732" path="/var/lib/kubelet/pods/b428ad4f-cd43-4daa-8f01-e94c7ef63732/volumes"
Oct 31 02:00:24.164766 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:24.164733    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:24.166090 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:24.166061    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:00:26.188636 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:26.188601    5650 generic.go:356] "Generic (PLEG): container finished" podID="1da683f3-ae35-4c60-a119-7711ca882f54" containerID="06dcb568530b690aec267324abd0782c765b29d05cd80cab6a092952e8877d97" exitCode=0
Oct 31 02:00:26.189559 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:26.188678    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww" event={"ID":"1da683f3-ae35-4c60-a119-7711ca882f54","Type":"ContainerDied","Data":"06dcb568530b690aec267324abd0782c765b29d05cd80cab6a092952e8877d97"}
Oct 31 02:00:26.189559 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:26.189284    5650 scope.go:122] "RemoveContainer" containerID="06dcb568530b690aec267324abd0782c765b29d05cd80cab6a092952e8877d97"
Oct 31 02:00:26.193793 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:26.193752    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:00:26.193939 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:26.193851    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:00:26.193939 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:26.193897    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:00:27.406088 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:27.406045    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww" event={"ID":"1da683f3-ae35-4c60-a119-7711ca882f54","Type":"ContainerStarted","Data":"262ddd17bef1830147e301ee710a84267ed9f12c6375813f645e24ceef2d76b5"}
Oct 31 02:00:27.839362 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:27.838911    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd"
Oct 31 02:00:27.839602 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:27.839410    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c"
Oct 31 02:00:27.861207 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:27.861170    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 02:00:27.862064 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:27.861630    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 31 02:00:27.862064 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:27.861697    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww" podUID="1da683f3-ae35-4c60-a119-7711ca882f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 31 02:00:28.604637 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:00:28.604595    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:00:28.861229 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:28.861142    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 31 02:00:28.861229 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:28.861208    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww" podUID="1da683f3-ae35-4c60-a119-7711ca882f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 31 02:00:29.032701 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:29.032666    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddba8822-rcwh7"]
Oct 31 02:00:29.037192 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:29.037164    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddba8822-rcwh7"]
Oct 31 02:00:29.941502 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:29.941408    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 31 02:00:29.942046 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:29.941497    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww" podUID="1da683f3-ae35-4c60-a119-7711ca882f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 31 02:00:30.135243 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:30.135207    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e714d0b9-3aa2-4cce-9b1c-1e2f543164a6" path="/var/lib/kubelet/pods/e714d0b9-3aa2-4cce-9b1c-1e2f543164a6/volumes"
Oct 31 02:00:30.862991 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:30.862937    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 31 02:00:30.863341 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:30.863300    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww" podUID="1da683f3-ae35-4c60-a119-7711ca882f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 31 02:00:31.861594 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:31.861553    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 31 02:00:31.862083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:31.861630    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww" podUID="1da683f3-ae35-4c60-a119-7711ca882f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 31 02:00:32.861834 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:32.861795    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 31 02:00:32.862275 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:32.861857    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww" podUID="1da683f3-ae35-4c60-a119-7711ca882f54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 31 02:00:33.867912 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:33.867880    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 02:00:33.868411 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:33.868325    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 02:00:33.869299 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:33.869279    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 02:00:45.584193 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.584153    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"]
Oct 31 02:00:45.585225 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585198    5650 cpu_manager.go:414] "containerMap: removing container" podUID="b428ad4f-cd43-4daa-8f01-e94c7ef63732" containerName="cleanup"
Oct 31 02:00:45.585225 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585224    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b428ad4f-cd43-4daa-8f01-e94c7ef63732" containerName="cleanup"
Oct 31 02:00:45.585405 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585252    5650 cpu_manager.go:414] "containerMap: removing container" podUID="0c56489d-d1a0-4ae3-9260-eb0d18201d12" containerName="cleanup"
Oct 31 02:00:45.585405 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585259    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c56489d-d1a0-4ae3-9260-eb0d18201d12" containerName="cleanup"
Oct 31 02:00:45.585405 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585280    5650 cpu_manager.go:414] "containerMap: removing container" podUID="e28001ed-ca1a-4790-96f5-73ada860e6de" containerName="cleanup"
Oct 31 02:00:45.585405 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585289    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e28001ed-ca1a-4790-96f5-73ada860e6de" containerName="cleanup"
Oct 31 02:00:45.585405 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585306    5650 cpu_manager.go:414] "containerMap: removing container" podUID="2bcc823c-fc5f-47c0-9cfa-665f05c71056" containerName="cleanup"
Oct 31 02:00:45.585405 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585313    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bcc823c-fc5f-47c0-9cfa-665f05c71056" containerName="cleanup"
Oct 31 02:00:45.585405 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585328    5650 cpu_manager.go:414] "containerMap: removing container" podUID="0c56489d-d1a0-4ae3-9260-eb0d18201d12" containerName="cleanup"
Oct 31 02:00:45.585405 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585335    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c56489d-d1a0-4ae3-9260-eb0d18201d12" containerName="cleanup"
Oct 31 02:00:45.585405 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585346    5650 cpu_manager.go:414] "containerMap: removing container" podUID="0c56489d-d1a0-4ae3-9260-eb0d18201d12" containerName="cleanup"
Oct 31 02:00:45.585405 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585353    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c56489d-d1a0-4ae3-9260-eb0d18201d12" containerName="cleanup"
Oct 31 02:00:45.585405 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585394    5650 cpu_manager.go:414] "containerMap: removing container" podUID="b428ad4f-cd43-4daa-8f01-e94c7ef63732" containerName="cleanup"
Oct 31 02:00:45.585405 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585401    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b428ad4f-cd43-4daa-8f01-e94c7ef63732" containerName="cleanup"
Oct 31 02:00:45.586066 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585415    5650 cpu_manager.go:414] "containerMap: removing container" podUID="2bcc823c-fc5f-47c0-9cfa-665f05c71056" containerName="cleanup"
Oct 31 02:00:45.586066 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585422    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bcc823c-fc5f-47c0-9cfa-665f05c71056" containerName="cleanup"
Oct 31 02:00:45.586066 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585457    5650 cpu_manager.go:414] "containerMap: removing container" podUID="c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d" containerName="cleanup"
Oct 31 02:00:45.586066 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585464    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d" containerName="cleanup"
Oct 31 02:00:45.586066 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585499    5650 cpu_manager.go:414] "containerMap: removing container" podUID="d87648fb-f2fb-42b8-b36b-22381fac5d7d" containerName="cleanup"
Oct 31 02:00:45.586066 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585506    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d87648fb-f2fb-42b8-b36b-22381fac5d7d" containerName="cleanup"
Oct 31 02:00:45.586066 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585722    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="b428ad4f-cd43-4daa-8f01-e94c7ef63732" containerName="cleanup"
Oct 31 02:00:45.586066 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585738    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="b428ad4f-cd43-4daa-8f01-e94c7ef63732" containerName="cleanup"
Oct 31 02:00:45.586066 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585751    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53f7ff7-14b2-4cbd-a7ba-0e848e2e7f1d" containerName="cleanup"
Oct 31 02:00:45.586066 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585762    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c56489d-d1a0-4ae3-9260-eb0d18201d12" containerName="cleanup"
Oct 31 02:00:45.586066 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585774    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bcc823c-fc5f-47c0-9cfa-665f05c71056" containerName="cleanup"
Oct 31 02:00:45.586066 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585784    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f86643f-3419-470f-a3e9-4aae61fc266c" containerName="cleanup"
Oct 31 02:00:45.586066 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585795    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c56489d-d1a0-4ae3-9260-eb0d18201d12" containerName="cleanup"
Oct 31 02:00:45.586066 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.585807    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bcc823c-fc5f-47c0-9cfa-665f05c71056" containerName="cleanup"
Oct 31 02:00:45.593243 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.593204    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.604820 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.604784    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-2186c3-openshift-default-dockercfg-pdnvx\""
Oct 31 02:00:45.608171 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.608136    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"]
Oct 31 02:00:45.714462 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.714409    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-podinfo\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.714657 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.714561    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-token\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.714657 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.714635    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-workload-socket\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.714745 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.714666    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-workload-certs\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.714812 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.714799    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-envoy\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.714872 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.714858    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-credential-socket\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.714999 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.714985    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-data\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.715105 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.715092    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istiod-ca-cert\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.715158 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.715146    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j59rj\" (UniqueName: \"kubernetes.io/projected/cd4d7677-6d53-42e9-8416-024a78dc3ff7-kube-api-access-j59rj\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.827485 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.821772    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-envoy\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.827485 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.821888    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-credential-socket\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.827485 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.822047    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-data\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.827485 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.822188    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istiod-ca-cert\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.827485 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.822237    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j59rj\" (UniqueName: \"kubernetes.io/projected/cd4d7677-6d53-42e9-8416-024a78dc3ff7-kube-api-access-j59rj\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.827485 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.822352    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-podinfo\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.827485 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.822385    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-token\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.827485 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.822418    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-workload-socket\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.827485 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.822464    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-workload-certs\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.827485 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.823071    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-workload-certs\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.839520 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.837877    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-credential-socket\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.839520 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.838154    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-data\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.839520 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.838832    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istiod-ca-cert\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.843474 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.841167    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-workload-socket\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.851477 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.849795    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-envoy\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.873473 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.868078    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-podinfo\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.907468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.905336    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j59rj\" (UniqueName: \"kubernetes.io/projected/cd4d7677-6d53-42e9-8416-024a78dc3ff7-kube-api-access-j59rj\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.907468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.906044    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-token\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.940151 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.938361    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:45.957471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:45.945172    5650 scope.go:122] "RemoveContainer" containerID="88212fe5ef44b7ec845c18c2ad65e53910b67d16826ddbb662f383b47d0bac8a"
Oct 31 02:00:46.079156 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:46.077701    5650 scope.go:122] "RemoveContainer" containerID="8ae041faa12b5b9f08de5ae0c877f165b348e7e7ae8f3f8ecd0f4a316e1681c6"
Oct 31 02:00:46.104595 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:46.104561    5650 scope.go:122] "RemoveContainer" containerID="07cc2ab40944aadf5b143ff44eb044cd0569d2851ee3591b04dd41190f349d4d"
Oct 31 02:00:46.384987 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:00:46.384954    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd4d7677_6d53_42e9_8416_024a78dc3ff7.slice/crio-6d5e18e2cdabd93bc5b89399d6f22a413ada0a16b733a9fc2ca412ebade18ccb WatchSource:0}: Error finding container 6d5e18e2cdabd93bc5b89399d6f22a413ada0a16b733a9fc2ca412ebade18ccb: Status 404 returned error can't find the container with id 6d5e18e2cdabd93bc5b89399d6f22a413ada0a16b733a9fc2ca412ebade18ccb
Oct 31 02:00:46.385930 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:46.385901    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"]
Oct 31 02:00:46.388207 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:46.388171    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:00:46.388309 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:46.388241    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:00:46.388309 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:46.388273    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:00:46.721032 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:46.720996    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg" event={"ID":"cd4d7677-6d53-42e9-8416-024a78dc3ff7","Type":"ContainerStarted","Data":"59dc66fa26cf1f535189f4243a938a8ae24805dfaa3854719c2693b9ee087812"}
Oct 31 02:00:46.721633 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:46.721055    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg" event={"ID":"cd4d7677-6d53-42e9-8416-024a78dc3ff7","Type":"ContainerStarted","Data":"6d5e18e2cdabd93bc5b89399d6f22a413ada0a16b733a9fc2ca412ebade18ccb"}
Oct 31 02:00:46.753155 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:46.753066    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg" podStartSLOduration=1.753043514 podStartE2EDuration="1.753043514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:46.748968938 +0000 UTC m=+2402.752683230" watchObservedRunningTime="2026-10-31 02:00:46.753043514 +0000 UTC m=+2402.756757807"
Oct 31 02:00:46.947405 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:46.947371    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:47.953190 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:47.953149    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:48.738590 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:48.738555    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:48.739782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:48.739761    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:00:52.988397 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:52.988362    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7"]
Oct 31 02:00:52.989577 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:52.989551    5650 cpu_manager.go:414] "containerMap: removing container" podUID="2bcc823c-fc5f-47c0-9cfa-665f05c71056" containerName="cleanup"
Oct 31 02:00:52.989577 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:52.989576    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bcc823c-fc5f-47c0-9cfa-665f05c71056" containerName="cleanup"
Oct 31 02:00:52.989787 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:52.989643    5650 cpu_manager.go:414] "containerMap: removing container" podUID="b428ad4f-cd43-4daa-8f01-e94c7ef63732" containerName="cleanup"
Oct 31 02:00:52.989787 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:52.989653    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b428ad4f-cd43-4daa-8f01-e94c7ef63732" containerName="cleanup"
Oct 31 02:00:52.989889 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:52.989860    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="b428ad4f-cd43-4daa-8f01-e94c7ef63732" containerName="cleanup"
Oct 31 02:00:52.989889 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:52.989881    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c56489d-d1a0-4ae3-9260-eb0d18201d12" containerName="cleanup"
Oct 31 02:00:52.989999 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:52.989892    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bcc823c-fc5f-47c0-9cfa-665f05c71056" containerName="cleanup"
Oct 31 02:00:52.994654 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:52.994624    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7"
Oct 31 02:00:53.005087 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:53.005052    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d0c6ff\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d0c6ff\""
Oct 31 02:00:53.005339 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:53.005314    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d0c6ff\"/\"payload-processing-external-model-e2e--4f663f8f-dockercfg-k6x5j\""
Oct 31 02:00:53.005411 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:53.005401    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d0c6ff\"/\"agc-cred-28829414036f\""
Oct 31 02:00:53.005546 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:53.005530    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d0c6ff\"/\"openshift-service-ca.crt\""
Oct 31 02:00:53.005623 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:53.005581    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d0c6ff\"/\"routing-overlay\""
Oct 31 02:00:53.005682 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:53.005643    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d0c6ff\"/\"kube-root-ca.crt\""
Oct 31 02:00:53.006788 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:53.006762    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppbhc\" (UniqueName: \"kubernetes.io/projected/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-kube-api-access-ppbhc\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7\" (UID: \"bdb97f84-e14d-4bfc-a5ff-7b2041607ef8\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7"
Oct 31 02:00:53.006946 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:53.006931    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7\" (UID: \"bdb97f84-e14d-4bfc-a5ff-7b2041607ef8\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7"
Oct 31 02:00:53.007046 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:53.007034    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7\" (UID: \"bdb97f84-e14d-4bfc-a5ff-7b2041607ef8\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7"
Oct 31 02:00:53.007114 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:53.007105    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7\" (UID: \"bdb97f84-e14d-4bfc-a5ff-7b2041607ef8\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7"
Oct 31 02:00:53.012858 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:53.010874    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7"]
Oct 31 02:00:53.121467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:53.120883    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7\" (UID: \"bdb97f84-e14d-4bfc-a5ff-7b2041607ef8\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7"
Oct 31 02:00:53.121467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:53.121011    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7\" (UID: \"bdb97f84-e14d-4bfc-a5ff-7b2041607ef8\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7"
Oct 31 02:00:53.121467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:53.121103    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7\" (UID: \"bdb97f84-e14d-4bfc-a5ff-7b2041607ef8\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7"
Oct 31 02:00:53.121467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:53.121341    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppbhc\" (UniqueName: \"kubernetes.io/projected/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-kube-api-access-ppbhc\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7\" (UID: \"bdb97f84-e14d-4bfc-a5ff-7b2041607ef8\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7"
Oct 31 02:00:53.167031 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:53.166995    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7\" (UID: \"bdb97f84-e14d-4bfc-a5ff-7b2041607ef8\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7"
Oct 31 02:00:53.209620 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:53.187421    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7\" (UID: \"bdb97f84-e14d-4bfc-a5ff-7b2041607ef8\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7"
Oct 31 02:00:53.240347 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:53.240172    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7\" (UID: \"bdb97f84-e14d-4bfc-a5ff-7b2041607ef8\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7"
Oct 31 02:00:53.252500 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:53.252442    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppbhc\" (UniqueName: \"kubernetes.io/projected/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-kube-api-access-ppbhc\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7\" (UID: \"bdb97f84-e14d-4bfc-a5ff-7b2041607ef8\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7"
Oct 31 02:00:53.330582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:53.330428    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7"
Oct 31 02:00:53.713075 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:00:53.713023    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdb97f84_e14d_4bfc_a5ff_7b2041607ef8.slice/crio-019f7d43a691885c92db2ca1b9f7307252090e07bb117177363f3b31e1dbe535 WatchSource:0}: Error finding container 019f7d43a691885c92db2ca1b9f7307252090e07bb117177363f3b31e1dbe535: Status 404 returned error can't find the container with id 019f7d43a691885c92db2ca1b9f7307252090e07bb117177363f3b31e1dbe535
Oct 31 02:00:53.713393 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:53.713367    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7"]
Oct 31 02:00:53.994528 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:53.994402    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7" event={"ID":"bdb97f84-e14d-4bfc-a5ff-7b2041607ef8","Type":"ContainerStarted","Data":"9d837ed1691b08b493fefe7a3f98002006d77fabcc2dd1694d0dae2dfc0e7d41"}
Oct 31 02:00:53.995500 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:53.994558    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7" event={"ID":"bdb97f84-e14d-4bfc-a5ff-7b2041607ef8","Type":"ContainerStarted","Data":"019f7d43a691885c92db2ca1b9f7307252090e07bb117177363f3b31e1dbe535"}
Oct 31 02:00:53.995500 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:53.994668    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7"
Oct 31 02:00:54.018510 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:54.018396    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7" podStartSLOduration=2.018379245 podStartE2EDuration="2.018379245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:54.017677752 +0000 UTC m=+2410.021392044" watchObservedRunningTime="2026-10-31 02:00:54.018379245 +0000 UTC m=+2410.022093536"
Oct 31 02:00:55.450430 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:55.450385    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"]
Oct 31 02:00:55.450946 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:00:55.450684    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6" podUID="c7b45430-8687-4cbb-b68c-f67723dfa9f6" containerName="istio-proxy" containerID="cri-o://db76364024cfe8bf5c03700d35c05029cbd2dc9a1b8b15107ec5cfb4bbaf8ec1" gracePeriod=30
Oct 31 02:00:58.753342 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:00:58.753298    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:01:00.749261 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.749189    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 02:01:00.892843 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.892802    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-podinfo\") pod \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") "
Oct 31 02:01:00.893047 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.892937    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-data\") pod \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") "
Oct 31 02:01:00.893171 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.893134    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-credential-socket\") pod \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") "
Oct 31 02:01:00.893248 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.893174    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-data" pod "c7b45430-8687-4cbb-b68c-f67723dfa9f6" (UID: "c7b45430-8687-4cbb-b68c-f67723dfa9f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:00.893248 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.893184    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istiod-ca-cert\") pod \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") "
Oct 31 02:01:00.893346 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.893291    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b45430-8687-4cbb-b68c-f67723dfa9f6-kube-api-access-4ktrt\" (UniqueName: \"kubernetes.io/projected/c7b45430-8687-4cbb-b68c-f67723dfa9f6-kube-api-access-4ktrt\") pod \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") "
Oct 31 02:01:00.893393 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.893346    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-workload-certs\") pod \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") "
Oct 31 02:01:00.893466 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.893397    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-credential-socket" pod "c7b45430-8687-4cbb-b68c-f67723dfa9f6" (UID: "c7b45430-8687-4cbb-b68c-f67723dfa9f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:00.893466 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.893402    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-token\" (UniqueName: \"kubernetes.io/projected/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-token\") pod \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") "
Oct 31 02:01:00.893590 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.893522    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-envoy\") pod \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") "
Oct 31 02:01:00.893590 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.893579    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istiod-ca-cert" pod "c7b45430-8687-4cbb-b68c-f67723dfa9f6" (UID: "c7b45430-8687-4cbb-b68c-f67723dfa9f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:01:00.893714 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.893669    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-workload-certs" pod "c7b45430-8687-4cbb-b68c-f67723dfa9f6" (UID: "c7b45430-8687-4cbb-b68c-f67723dfa9f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:00.893714 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.893680    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-workload-socket\") pod \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\" (UID: \"c7b45430-8687-4cbb-b68c-f67723dfa9f6\") "
Oct 31 02:01:00.893975 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.893949    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-workload-socket" pod "c7b45430-8687-4cbb-b68c-f67723dfa9f6" (UID: "c7b45430-8687-4cbb-b68c-f67723dfa9f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:00.894485 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.894427    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:00.894594 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.894493    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:00.894594 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.894510    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:00.894594 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.894522    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:00.894594 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.894535    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:00.895642 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.895620    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-envoy" pod "c7b45430-8687-4cbb-b68c-f67723dfa9f6" (UID: "c7b45430-8687-4cbb-b68c-f67723dfa9f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:00.895748 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.895621    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-token" pod "c7b45430-8687-4cbb-b68c-f67723dfa9f6" (UID: "c7b45430-8687-4cbb-b68c-f67723dfa9f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:01:00.895815 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.895749    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-podinfo" pod "c7b45430-8687-4cbb-b68c-f67723dfa9f6" (UID: "c7b45430-8687-4cbb-b68c-f67723dfa9f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:01:00.895815 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.895762    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b45430-8687-4cbb-b68c-f67723dfa9f6-kube-api-access-4ktrt" pod "c7b45430-8687-4cbb-b68c-f67723dfa9f6" (UID: "c7b45430-8687-4cbb-b68c-f67723dfa9f6"). InnerVolumeSpecName "kube-api-access-4ktrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:01:00.996025 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.995989    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:00.996025 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.996021    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:00.996025 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.996033    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ktrt\" (UniqueName: \"kubernetes.io/projected/c7b45430-8687-4cbb-b68c-f67723dfa9f6-kube-api-access-4ktrt\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:00.996275 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:00.996043    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b45430-8687-4cbb-b68c-f67723dfa9f6-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:01.097062 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:01.096967    5650 generic.go:356] "Generic (PLEG): container finished" podID="c7b45430-8687-4cbb-b68c-f67723dfa9f6" containerID="db76364024cfe8bf5c03700d35c05029cbd2dc9a1b8b15107ec5cfb4bbaf8ec1" exitCode=0
Oct 31 02:01:01.097062 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:01.097013    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6" event={"ID":"c7b45430-8687-4cbb-b68c-f67723dfa9f6","Type":"ContainerDied","Data":"db76364024cfe8bf5c03700d35c05029cbd2dc9a1b8b15107ec5cfb4bbaf8ec1"}
Oct 31 02:01:01.097062 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:01.097045    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"
Oct 31 02:01:01.097062 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:01.097063    5650 scope.go:122] "RemoveContainer" containerID="db76364024cfe8bf5c03700d35c05029cbd2dc9a1b8b15107ec5cfb4bbaf8ec1"
Oct 31 02:01:01.097424 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:01.097054    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6" event={"ID":"c7b45430-8687-4cbb-b68c-f67723dfa9f6","Type":"ContainerDied","Data":"861de6decb9994a7ac3e1501db2cdc119ed22572dd40693c13a05748a17f054d"}
Oct 31 02:01:01.110557 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:01.110536    5650 scope.go:122] "RemoveContainer" containerID="db76364024cfe8bf5c03700d35c05029cbd2dc9a1b8b15107ec5cfb4bbaf8ec1"
Oct 31 02:01:01.110877 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:01:01.110853    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db76364024cfe8bf5c03700d35c05029cbd2dc9a1b8b15107ec5cfb4bbaf8ec1\": container with ID starting with db76364024cfe8bf5c03700d35c05029cbd2dc9a1b8b15107ec5cfb4bbaf8ec1 not found: ID does not exist" containerID="db76364024cfe8bf5c03700d35c05029cbd2dc9a1b8b15107ec5cfb4bbaf8ec1"
Oct 31 02:01:01.110948 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:01.110889    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db76364024cfe8bf5c03700d35c05029cbd2dc9a1b8b15107ec5cfb4bbaf8ec1"} err="failed to get container status \"db76364024cfe8bf5c03700d35c05029cbd2dc9a1b8b15107ec5cfb4bbaf8ec1\": rpc error: code = NotFound desc = could not find container \"db76364024cfe8bf5c03700d35c05029cbd2dc9a1b8b15107ec5cfb4bbaf8ec1\": container with ID starting with db76364024cfe8bf5c03700d35c05029cbd2dc9a1b8b15107ec5cfb4bbaf8ec1 not found: ID does not exist"
Oct 31 02:01:01.127835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:01.127678    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"]
Oct 31 02:01:01.132775 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:01.132749    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7hqq6"]
Oct 31 02:01:02.122350 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:02.122315    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b45430-8687-4cbb-b68c-f67723dfa9f6" path="/var/lib/kubelet/pods/c7b45430-8687-4cbb-b68c-f67723dfa9f6/volumes"
Oct 31 02:01:02.996196 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:02.996158    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"]
Oct 31 02:01:02.997606 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:02.997579    5650 cpu_manager.go:414] "containerMap: removing container" podUID="c7b45430-8687-4cbb-b68c-f67723dfa9f6" containerName="istio-proxy"
Oct 31 02:01:02.997606 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:02.997607    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b45430-8687-4cbb-b68c-f67723dfa9f6" containerName="istio-proxy"
Oct 31 02:01:02.997864 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:02.997847    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b45430-8687-4cbb-b68c-f67723dfa9f6" containerName="istio-proxy"
Oct 31 02:01:03.004164 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.004137    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.013357 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.013327    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-91b6b1-openshift-default-dockercfg-9wq4c\""
Oct 31 02:01:03.020147 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.020118    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"]
Oct 31 02:01:03.239713 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.228869    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42b42\" (UniqueName: \"kubernetes.io/projected/c060bf51-21db-4f5b-889d-e655338a9bfa-kube-api-access-42b42\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.240410 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.240373    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c060bf51-21db-4f5b-889d-e655338a9bfa-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.240756 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.240734    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.241010 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.240992    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.241639 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.241616    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.241848 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.241831    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.242318 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.242300    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.250527 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.246785    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.293471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.292901    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.410899 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.410856    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.421471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.411242    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42b42\" (UniqueName: \"kubernetes.io/projected/c060bf51-21db-4f5b-889d-e655338a9bfa-kube-api-access-42b42\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.434479 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.428607    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c060bf51-21db-4f5b-889d-e655338a9bfa-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.434479 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.428751    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.434479 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.428831    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.434479 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.428876    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.434479 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.428930    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.434479 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.429002    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.434479 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.429180    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.434479 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.430604    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.434479 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.431435    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c060bf51-21db-4f5b-889d-e655338a9bfa-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.434479 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.431724    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.434479 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.431948    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.434479 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.432189    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.458468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.456071    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42b42\" (UniqueName: \"kubernetes.io/projected/c060bf51-21db-4f5b-889d-e655338a9bfa-kube-api-access-42b42\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.469469 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.460904    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.469469 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.463819    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.471577 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.471504    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.624238 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.624131    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:03.932706 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:01:03.932672    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc060bf51_21db_4f5b_889d_e655338a9bfa.slice/crio-a48695e28a959d67626f196ff8187a55f9ecf96f52fcb2e295cfcff12d7a17bb WatchSource:0}: Error finding container a48695e28a959d67626f196ff8187a55f9ecf96f52fcb2e295cfcff12d7a17bb: Status 404 returned error can't find the container with id a48695e28a959d67626f196ff8187a55f9ecf96f52fcb2e295cfcff12d7a17bb
Oct 31 02:01:03.932915 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.932891    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"]
Oct 31 02:01:03.936293 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.936254    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:01:03.936414 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.936341    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:01:03.936414 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:03.936388    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:01:04.134174 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:04.134137    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc" event={"ID":"c060bf51-21db-4f5b-889d-e655338a9bfa","Type":"ContainerStarted","Data":"749f9e14419765c55ba413d5e5d93be4f909775c40d3ecb3b30a3d533f6b4c57"}
Oct 31 02:01:04.134380 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:04.134198    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc" event={"ID":"c060bf51-21db-4f5b-889d-e655338a9bfa","Type":"ContainerStarted","Data":"a48695e28a959d67626f196ff8187a55f9ecf96f52fcb2e295cfcff12d7a17bb"}
Oct 31 02:01:04.173474 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:04.172627    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc" podStartSLOduration=2.17260412 podStartE2EDuration="2.17260412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:04.170103903 +0000 UTC m=+2420.173818196" watchObservedRunningTime="2026-10-31 02:01:04.17260412 +0000 UTC m=+2420.176318412"
Oct 31 02:01:04.624764 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:04.624727    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:05.007464 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:05.007421    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7"
Oct 31 02:01:05.693473 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:05.691986    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:06.163185 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:06.163148    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:06.164414 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:06.164392    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:01:07.950359 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:07.950318    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g"]
Oct 31 02:01:07.955651 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:07.955623    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g"
Oct 31 02:01:07.967587 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:07.967554    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-91b6b1-dockercfg-vszh2\""
Oct 31 02:01:07.969619 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:07.969592    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-91b6b1\""
Oct 31 02:01:07.971548 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:07.971523    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g"]
Oct 31 02:01:08.037154 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:08.037105    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28ml8\" (UniqueName: \"kubernetes.io/projected/44fe8812-7863-4904-bb34-5ee90f27db75-kube-api-access-28ml8\") pod \"payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g\" (UID: \"44fe8812-7863-4904-bb34-5ee90f27db75\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g"
Oct 31 02:01:08.037365 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:08.037240    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44fe8812-7863-4904-bb34-5ee90f27db75-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g\" (UID: \"44fe8812-7863-4904-bb34-5ee90f27db75\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g"
Oct 31 02:01:08.197979 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:08.197930    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28ml8\" (UniqueName: \"kubernetes.io/projected/44fe8812-7863-4904-bb34-5ee90f27db75-kube-api-access-28ml8\") pod \"payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g\" (UID: \"44fe8812-7863-4904-bb34-5ee90f27db75\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g"
Oct 31 02:01:08.198266 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:08.198246    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44fe8812-7863-4904-bb34-5ee90f27db75-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g\" (UID: \"44fe8812-7863-4904-bb34-5ee90f27db75\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g"
Oct 31 02:01:08.199346 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:08.199320    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44fe8812-7863-4904-bb34-5ee90f27db75-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g\" (UID: \"44fe8812-7863-4904-bb34-5ee90f27db75\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g"
Oct 31 02:01:08.388809 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:08.388772    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28ml8\" (UniqueName: \"kubernetes.io/projected/44fe8812-7863-4904-bb34-5ee90f27db75-kube-api-access-28ml8\") pod \"payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g\" (UID: \"44fe8812-7863-4904-bb34-5ee90f27db75\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g"
Oct 31 02:01:08.574942 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:08.574900    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g"
Oct 31 02:01:08.895057 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:01:08.895024    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44fe8812_7863_4904_bb34_5ee90f27db75.slice/crio-97d93a372b11cfc3839c70f1a597ab809171e6eb9984b9addbec7f367382bd89 WatchSource:0}: Error finding container 97d93a372b11cfc3839c70f1a597ab809171e6eb9984b9addbec7f367382bd89: Status 404 returned error can't find the container with id 97d93a372b11cfc3839c70f1a597ab809171e6eb9984b9addbec7f367382bd89
Oct 31 02:01:08.895874 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:08.895846    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g"]
Oct 31 02:01:09.264499 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.264414    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"]
Oct 31 02:01:09.269792 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.269765    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.285595 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.284231    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"]
Oct 31 02:01:09.285595 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.284927    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-token\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.285595 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.285258    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-workload-certs\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.285595 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.285297    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-workload-socket\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.285595 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.285401    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-credential-socket\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.285595 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.285439    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-data\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.285595 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.285477    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-podinfo\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.286106 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.285538    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-envoy\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.286106 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.285666    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5edd74fe-6362-4479-b949-c44183eb2b0d-istiod-ca-cert\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.286106 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.285701    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47znk\" (UniqueName: \"kubernetes.io/projected/5edd74fe-6362-4479-b949-c44183eb2b0d-kube-api-access-47znk\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.374601 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.374571    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-d0c6ff-openshift-default-dockercfg-j64m7\""
Oct 31 02:01:09.388154 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.388119    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-workload-certs\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.388357 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.388169    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-workload-socket\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.388357 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.388248    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-credential-socket\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.388357 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.388283    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-data\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.388357 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.388309    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-podinfo\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.388666 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.388367    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-envoy\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.388666 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.388411    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5edd74fe-6362-4479-b949-c44183eb2b0d-istiod-ca-cert\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.388666 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.388460    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47znk\" (UniqueName: \"kubernetes.io/projected/5edd74fe-6362-4479-b949-c44183eb2b0d-kube-api-access-47znk\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.388666 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.388586    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-workload-certs\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.388666 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.388638    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-workload-socket\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.388995 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.388765    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-token\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.389267 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.389244    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-credential-socket\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.389399 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.389320    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5edd74fe-6362-4479-b949-c44183eb2b0d-istiod-ca-cert\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.389938 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.389910    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-data\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.392066 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.392039    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-envoy\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.392470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.392429    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-podinfo\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.401037 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.401006    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g" event={"ID":"44fe8812-7863-4904-bb34-5ee90f27db75","Type":"ContainerStarted","Data":"49bbaf641308eeb3b1a50644103c0959e57337e28375476c0b612858d26e3bd5"}
Oct 31 02:01:09.401216 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.401059    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g" event={"ID":"44fe8812-7863-4904-bb34-5ee90f27db75","Type":"ContainerStarted","Data":"97d93a372b11cfc3839c70f1a597ab809171e6eb9984b9addbec7f367382bd89"}
Oct 31 02:01:09.401216 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.401089    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g"
Oct 31 02:01:09.403210 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.403185    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-token\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.403425 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.403402    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47znk\" (UniqueName: \"kubernetes.io/projected/5edd74fe-6362-4479-b949-c44183eb2b0d-kube-api-access-47znk\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.422734 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.422678    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g" podStartSLOduration=2.422662889 podStartE2EDuration="2.422662889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:09.421628568 +0000 UTC m=+2425.425342858" watchObservedRunningTime="2026-10-31 02:01:09.422662889 +0000 UTC m=+2425.426377180"
Oct 31 02:01:09.594702 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.594311    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:09.906599 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:01:09.906551    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5edd74fe_6362_4479_b949_c44183eb2b0d.slice/crio-7f17920f3522889f107426af5bd536dd8ed25ab0757528a956cdd36f778a003b WatchSource:0}: Error finding container 7f17920f3522889f107426af5bd536dd8ed25ab0757528a956cdd36f778a003b: Status 404 returned error can't find the container with id 7f17920f3522889f107426af5bd536dd8ed25ab0757528a956cdd36f778a003b
Oct 31 02:01:09.907209 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.907181    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"]
Oct 31 02:01:09.910836 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.910795    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:01:09.910955 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.910891    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:01:09.910955 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:09.910934    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:01:10.412505 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:10.412463    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q" event={"ID":"5edd74fe-6362-4479-b949-c44183eb2b0d","Type":"ContainerStarted","Data":"bf8d7c2b0239c4d535a26dfe1ad8b4e904f3b8344339efb667213066e153bab6"}
Oct 31 02:01:10.412935 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:10.412538    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q" event={"ID":"5edd74fe-6362-4479-b949-c44183eb2b0d","Type":"ContainerStarted","Data":"7f17920f3522889f107426af5bd536dd8ed25ab0757528a956cdd36f778a003b"}
Oct 31 02:01:10.447001 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:10.446937    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q" podStartSLOduration=1.446919224 podStartE2EDuration="1.446919224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:10.445197733 +0000 UTC m=+2426.448912029" watchObservedRunningTime="2026-10-31 02:01:10.446919224 +0000 UTC m=+2426.450633518"
Oct 31 02:01:10.595524 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:10.595485    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:10.602261 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:10.599625    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 31 02:01:10.602261 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:10.599692    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q" podUID="5edd74fe-6362-4479-b949-c44183eb2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 31 02:01:11.596111 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:11.596068    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 31 02:01:11.596550 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:11.596129    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q" podUID="5edd74fe-6362-4479-b949-c44183eb2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 31 02:01:11.844571 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:11.844530    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"]
Oct 31 02:01:11.844852 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:11.844820    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl" podUID="2d872f42-06a3-4580-8930-fbe87eac3311" containerName="istio-proxy" containerID="cri-o://68180ed21c305f7c9115e6be1961d5964b92606e157d41f5824f60ac86c14902" gracePeriod=30
Oct 31 02:01:12.595433 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:12.595400    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 31 02:01:12.595636 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:12.595468    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q" podUID="5edd74fe-6362-4479-b949-c44183eb2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 31 02:01:13.595648 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:13.595614    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 31 02:01:13.596096 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:13.595670    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q" podUID="5edd74fe-6362-4479-b949-c44183eb2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 31 02:01:14.595819 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:14.595785    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 31 02:01:14.596279 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:14.595843    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q" podUID="5edd74fe-6362-4479-b949-c44183eb2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 31 02:01:15.596115 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:15.596079    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 31 02:01:15.596663 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:15.596153    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q" podUID="5edd74fe-6362-4479-b949-c44183eb2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 31 02:01:16.601803 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:16.601762    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:16.602314 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:16.602272    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:16.603333 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:16.603307    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:01:17.190356 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.190327    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 02:01:17.314006 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.313966    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d872f42-06a3-4580-8930-fbe87eac3311-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d872f42-06a3-4580-8930-fbe87eac3311-istio-podinfo\") pod \"2d872f42-06a3-4580-8930-fbe87eac3311\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") "
Oct 31 02:01:17.314189 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.314071    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d872f42-06a3-4580-8930-fbe87eac3311-kube-api-access-p4fxk\" (UniqueName: \"kubernetes.io/projected/2d872f42-06a3-4580-8930-fbe87eac3311-kube-api-access-p4fxk\") pod \"2d872f42-06a3-4580-8930-fbe87eac3311\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") "
Oct 31 02:01:17.314256 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.314194    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-istio-envoy\") pod \"2d872f42-06a3-4580-8930-fbe87eac3311\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") "
Oct 31 02:01:17.314256 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.314247    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d872f42-06a3-4580-8930-fbe87eac3311-istio-token\" (UniqueName: \"kubernetes.io/projected/2d872f42-06a3-4580-8930-fbe87eac3311-istio-token\") pod \"2d872f42-06a3-4580-8930-fbe87eac3311\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") "
Oct 31 02:01:17.314392 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.314304    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-workload-socket\") pod \"2d872f42-06a3-4580-8930-fbe87eac3311\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") "
Oct 31 02:01:17.314504 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.314487    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-credential-socket\") pod \"2d872f42-06a3-4580-8930-fbe87eac3311\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") "
Oct 31 02:01:17.314572 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.314515    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-workload-certs\") pod \"2d872f42-06a3-4580-8930-fbe87eac3311\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") "
Oct 31 02:01:17.314696 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.314674    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-istio-data\") pod \"2d872f42-06a3-4580-8930-fbe87eac3311\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") "
Oct 31 02:01:17.314787 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.314685    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-workload-socket" pod "2d872f42-06a3-4580-8930-fbe87eac3311" (UID: "2d872f42-06a3-4580-8930-fbe87eac3311"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:17.314787 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.314720    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d872f42-06a3-4580-8930-fbe87eac3311-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d872f42-06a3-4580-8930-fbe87eac3311-istiod-ca-cert\") pod \"2d872f42-06a3-4580-8930-fbe87eac3311\" (UID: \"2d872f42-06a3-4580-8930-fbe87eac3311\") "
Oct 31 02:01:17.314931 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.314908    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-workload-certs" pod "2d872f42-06a3-4580-8930-fbe87eac3311" (UID: "2d872f42-06a3-4580-8930-fbe87eac3311"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:17.315144 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.315098    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-credential-socket" pod "2d872f42-06a3-4580-8930-fbe87eac3311" (UID: "2d872f42-06a3-4580-8930-fbe87eac3311"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:17.315293 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.315272    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-istio-data" pod "2d872f42-06a3-4580-8930-fbe87eac3311" (UID: "2d872f42-06a3-4580-8930-fbe87eac3311"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:17.315522 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.315501    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d872f42-06a3-4580-8930-fbe87eac3311-istiod-ca-cert" pod "2d872f42-06a3-4580-8930-fbe87eac3311" (UID: "2d872f42-06a3-4580-8930-fbe87eac3311"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:01:17.315869 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.315852    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:17.315955 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.315882    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:17.315955 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.315894    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:17.315955 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.315903    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:17.315955 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.315914    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d872f42-06a3-4580-8930-fbe87eac3311-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:17.316827 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.316805    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d872f42-06a3-4580-8930-fbe87eac3311-kube-api-access-p4fxk" pod "2d872f42-06a3-4580-8930-fbe87eac3311" (UID: "2d872f42-06a3-4580-8930-fbe87eac3311"). InnerVolumeSpecName "kube-api-access-p4fxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:01:17.316926 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.316825    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d872f42-06a3-4580-8930-fbe87eac3311-istio-podinfo" pod "2d872f42-06a3-4580-8930-fbe87eac3311" (UID: "2d872f42-06a3-4580-8930-fbe87eac3311"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:01:17.317015 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.316991    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d872f42-06a3-4580-8930-fbe87eac3311-istio-token" pod "2d872f42-06a3-4580-8930-fbe87eac3311" (UID: "2d872f42-06a3-4580-8930-fbe87eac3311"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:01:17.317055 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.317038    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-istio-envoy" pod "2d872f42-06a3-4580-8930-fbe87eac3311" (UID: "2d872f42-06a3-4580-8930-fbe87eac3311"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:17.418545 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.418460    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d872f42-06a3-4580-8930-fbe87eac3311-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:17.418545 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.418497    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4fxk\" (UniqueName: \"kubernetes.io/projected/2d872f42-06a3-4580-8930-fbe87eac3311-kube-api-access-p4fxk\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:17.418545 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.418512    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d872f42-06a3-4580-8930-fbe87eac3311-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:17.418545 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.418528    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d872f42-06a3-4580-8930-fbe87eac3311-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:17.510565 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.510529    5650 generic.go:356] "Generic (PLEG): container finished" podID="2d872f42-06a3-4580-8930-fbe87eac3311" containerID="68180ed21c305f7c9115e6be1961d5964b92606e157d41f5824f60ac86c14902" exitCode=0
Oct 31 02:01:17.510744 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.510591    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"
Oct 31 02:01:17.510744 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.510605    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl" event={"ID":"2d872f42-06a3-4580-8930-fbe87eac3311","Type":"ContainerDied","Data":"68180ed21c305f7c9115e6be1961d5964b92606e157d41f5824f60ac86c14902"}
Oct 31 02:01:17.510744 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.510661    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl" event={"ID":"2d872f42-06a3-4580-8930-fbe87eac3311","Type":"ContainerDied","Data":"8526a12bfc80aaded94216dd8a491039a2c64390a936f5290876ec23fc3155cf"}
Oct 31 02:01:17.510744 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.510686    5650 scope.go:122] "RemoveContainer" containerID="68180ed21c305f7c9115e6be1961d5964b92606e157d41f5824f60ac86c14902"
Oct 31 02:01:17.525641 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.525608    5650 scope.go:122] "RemoveContainer" containerID="68180ed21c305f7c9115e6be1961d5964b92606e157d41f5824f60ac86c14902"
Oct 31 02:01:17.525924 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:01:17.525899    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68180ed21c305f7c9115e6be1961d5964b92606e157d41f5824f60ac86c14902\": container with ID starting with 68180ed21c305f7c9115e6be1961d5964b92606e157d41f5824f60ac86c14902 not found: ID does not exist" containerID="68180ed21c305f7c9115e6be1961d5964b92606e157d41f5824f60ac86c14902"
Oct 31 02:01:17.526022 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.525930    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68180ed21c305f7c9115e6be1961d5964b92606e157d41f5824f60ac86c14902"} err="failed to get container status \"68180ed21c305f7c9115e6be1961d5964b92606e157d41f5824f60ac86c14902\": rpc error: code = NotFound desc = could not find container \"68180ed21c305f7c9115e6be1961d5964b92606e157d41f5824f60ac86c14902\": container with ID starting with 68180ed21c305f7c9115e6be1961d5964b92606e157d41f5824f60ac86c14902 not found: ID does not exist"
Oct 31 02:01:17.555876 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.555830    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"]
Oct 31 02:01:17.562350 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:17.562317    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ft8wl"]
Oct 31 02:01:18.121576 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:18.121539    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d872f42-06a3-4580-8930-fbe87eac3311" path="/var/lib/kubelet/pods/2d872f42-06a3-4580-8930-fbe87eac3311/volumes"
Oct 31 02:01:20.416388 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:20.416351    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g"
Oct 31 02:01:22.333403 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:22.333360    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"]
Oct 31 02:01:22.333910 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:22.333602    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx" podUID="b61e54f0-3295-4e02-aceb-2190f5b00181" containerName="istio-proxy" containerID="cri-o://51f48d9755b14f29591aa6392f7dba98fa71d562466cdf99fab2710729c084a0" gracePeriod=30
Oct 31 02:01:23.921467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:23.920704    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"]
Oct 31 02:01:27.227201 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.227109    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"]
Oct 31 02:01:27.227666 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.227370    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph" podUID="154899b1-075f-4d48-82c6-5c0fb76bff18" containerName="istio-proxy" containerID="cri-o://c64317c691c04b77ba04327184b72a3f6d2fdd6bfcc2d80fbf3ffefd3d1880f8" gracePeriod=30
Oct 31 02:01:27.659401 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.659370    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 02:01:27.667728 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.667694    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-workload-certs\") pod \"b61e54f0-3295-4e02-aceb-2190f5b00181\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") "
Oct 31 02:01:27.667933 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.667750    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-workload-socket\") pod \"b61e54f0-3295-4e02-aceb-2190f5b00181\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") "
Oct 31 02:01:27.667933 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.667849    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-token\" (UniqueName: \"kubernetes.io/projected/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-token\") pod \"b61e54f0-3295-4e02-aceb-2190f5b00181\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") "
Oct 31 02:01:27.668058 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.667940    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-credential-socket\") pod \"b61e54f0-3295-4e02-aceb-2190f5b00181\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") "
Oct 31 02:01:27.668058 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.667975    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-envoy\") pod \"b61e54f0-3295-4e02-aceb-2190f5b00181\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") "
Oct 31 02:01:27.668058 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.668042    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-workload-certs" pod "b61e54f0-3295-4e02-aceb-2190f5b00181" (UID: "b61e54f0-3295-4e02-aceb-2190f5b00181"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:27.668289 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.668072    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b61e54f0-3295-4e02-aceb-2190f5b00181-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61e54f0-3295-4e02-aceb-2190f5b00181-istiod-ca-cert\") pod \"b61e54f0-3295-4e02-aceb-2190f5b00181\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") "
Oct 31 02:01:27.668289 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.668124    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-workload-socket" pod "b61e54f0-3295-4e02-aceb-2190f5b00181" (UID: "b61e54f0-3295-4e02-aceb-2190f5b00181"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:27.668289 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.668219    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-podinfo\") pod \"b61e54f0-3295-4e02-aceb-2190f5b00181\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") "
Oct 31 02:01:27.668289 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.668233    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-credential-socket" pod "b61e54f0-3295-4e02-aceb-2190f5b00181" (UID: "b61e54f0-3295-4e02-aceb-2190f5b00181"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:27.668746 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.668354    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61e54f0-3295-4e02-aceb-2190f5b00181-kube-api-access-znjcp\" (UniqueName: \"kubernetes.io/projected/b61e54f0-3295-4e02-aceb-2190f5b00181-kube-api-access-znjcp\") pod \"b61e54f0-3295-4e02-aceb-2190f5b00181\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") "
Oct 31 02:01:27.668746 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.668507    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61e54f0-3295-4e02-aceb-2190f5b00181-istiod-ca-cert" pod "b61e54f0-3295-4e02-aceb-2190f5b00181" (UID: "b61e54f0-3295-4e02-aceb-2190f5b00181"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:01:27.668746 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.668570    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-data\") pod \"b61e54f0-3295-4e02-aceb-2190f5b00181\" (UID: \"b61e54f0-3295-4e02-aceb-2190f5b00181\") "
Oct 31 02:01:27.668913 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.668863    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-data" pod "b61e54f0-3295-4e02-aceb-2190f5b00181" (UID: "b61e54f0-3295-4e02-aceb-2190f5b00181"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:27.669597 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.669575    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:27.669720 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.669659    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61e54f0-3295-4e02-aceb-2190f5b00181-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:27.669720 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.669676    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:27.669720 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.669690    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:27.669720 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.669705    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:27.670835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.670811    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61e54f0-3295-4e02-aceb-2190f5b00181-kube-api-access-znjcp" pod "b61e54f0-3295-4e02-aceb-2190f5b00181" (UID: "b61e54f0-3295-4e02-aceb-2190f5b00181"). InnerVolumeSpecName "kube-api-access-znjcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:01:27.670975 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.670952    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-token" pod "b61e54f0-3295-4e02-aceb-2190f5b00181" (UID: "b61e54f0-3295-4e02-aceb-2190f5b00181"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:01:27.671044 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.670972    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-podinfo" pod "b61e54f0-3295-4e02-aceb-2190f5b00181" (UID: "b61e54f0-3295-4e02-aceb-2190f5b00181"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:01:27.671044 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.671005    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-envoy" pod "b61e54f0-3295-4e02-aceb-2190f5b00181" (UID: "b61e54f0-3295-4e02-aceb-2190f5b00181"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:27.708949 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.708909    5650 generic.go:356] "Generic (PLEG): container finished" podID="b61e54f0-3295-4e02-aceb-2190f5b00181" containerID="51f48d9755b14f29591aa6392f7dba98fa71d562466cdf99fab2710729c084a0" exitCode=0
Oct 31 02:01:27.709121 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.708961    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"
Oct 31 02:01:27.709121 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.708986    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx" event={"ID":"b61e54f0-3295-4e02-aceb-2190f5b00181","Type":"ContainerDied","Data":"51f48d9755b14f29591aa6392f7dba98fa71d562466cdf99fab2710729c084a0"}
Oct 31 02:01:27.709121 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.709041    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx" event={"ID":"b61e54f0-3295-4e02-aceb-2190f5b00181","Type":"ContainerDied","Data":"ae0d92e6b2bfa150f5e4e35a6f85a838f2e41c51d16001e3edfd432430df5832"}
Oct 31 02:01:27.709121 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.709068    5650 scope.go:122] "RemoveContainer" containerID="51f48d9755b14f29591aa6392f7dba98fa71d562466cdf99fab2710729c084a0"
Oct 31 02:01:27.723771 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.723741    5650 scope.go:122] "RemoveContainer" containerID="51f48d9755b14f29591aa6392f7dba98fa71d562466cdf99fab2710729c084a0"
Oct 31 02:01:27.724129 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:01:27.724106    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51f48d9755b14f29591aa6392f7dba98fa71d562466cdf99fab2710729c084a0\": container with ID starting with 51f48d9755b14f29591aa6392f7dba98fa71d562466cdf99fab2710729c084a0 not found: ID does not exist" containerID="51f48d9755b14f29591aa6392f7dba98fa71d562466cdf99fab2710729c084a0"
Oct 31 02:01:27.724242 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.724137    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51f48d9755b14f29591aa6392f7dba98fa71d562466cdf99fab2710729c084a0"} err="failed to get container status \"51f48d9755b14f29591aa6392f7dba98fa71d562466cdf99fab2710729c084a0\": rpc error: code = NotFound desc = could not find container \"51f48d9755b14f29591aa6392f7dba98fa71d562466cdf99fab2710729c084a0\": container with ID starting with 51f48d9755b14f29591aa6392f7dba98fa71d562466cdf99fab2710729c084a0 not found: ID does not exist"
Oct 31 02:01:27.745260 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.745214    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"]
Oct 31 02:01:27.751409 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.751375    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-fb8jx"]
Oct 31 02:01:27.772501 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.772419    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:27.772501 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.772469    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znjcp\" (UniqueName: \"kubernetes.io/projected/b61e54f0-3295-4e02-aceb-2190f5b00181-kube-api-access-znjcp\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:27.772501 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.772488    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:27.772501 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:27.772503    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61e54f0-3295-4e02-aceb-2190f5b00181-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:28.121103 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:28.121068    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b61e54f0-3295-4e02-aceb-2190f5b00181" path="/var/lib/kubelet/pods/b61e54f0-3295-4e02-aceb-2190f5b00181/volumes"
Oct 31 02:01:28.662470 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:01:28.659936    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:01:32.571624 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.571569    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 02:01:32.651603 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.651564    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-envoy\") pod \"154899b1-075f-4d48-82c6-5c0fb76bff18\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") "
Oct 31 02:01:32.651829 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.651626    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-credential-socket\") pod \"154899b1-075f-4d48-82c6-5c0fb76bff18\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") "
Oct 31 02:01:32.651829 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.651677    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/154899b1-075f-4d48-82c6-5c0fb76bff18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154899b1-075f-4d48-82c6-5c0fb76bff18-istiod-ca-cert\") pod \"154899b1-075f-4d48-82c6-5c0fb76bff18\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") "
Oct 31 02:01:32.651966 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.651845    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-workload-socket\") pod \"154899b1-075f-4d48-82c6-5c0fb76bff18\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") "
Oct 31 02:01:32.651966 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.651922    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154899b1-075f-4d48-82c6-5c0fb76bff18-kube-api-access-cqcbn\" (UniqueName: \"kubernetes.io/projected/154899b1-075f-4d48-82c6-5c0fb76bff18-kube-api-access-cqcbn\") pod \"154899b1-075f-4d48-82c6-5c0fb76bff18\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") "
Oct 31 02:01:32.652090 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.651995    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-token\" (UniqueName: \"kubernetes.io/projected/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-token\") pod \"154899b1-075f-4d48-82c6-5c0fb76bff18\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") "
Oct 31 02:01:32.652090 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.651999    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-credential-socket" pod "154899b1-075f-4d48-82c6-5c0fb76bff18" (UID: "154899b1-075f-4d48-82c6-5c0fb76bff18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:32.652090 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.652053    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-podinfo\") pod \"154899b1-075f-4d48-82c6-5c0fb76bff18\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") "
Oct 31 02:01:32.652257 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.652091    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-data\") pod \"154899b1-075f-4d48-82c6-5c0fb76bff18\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") "
Oct 31 02:01:32.652257 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.652141    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-workload-certs\") pod \"154899b1-075f-4d48-82c6-5c0fb76bff18\" (UID: \"154899b1-075f-4d48-82c6-5c0fb76bff18\") "
Oct 31 02:01:32.652257 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.652147    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-workload-socket" pod "154899b1-075f-4d48-82c6-5c0fb76bff18" (UID: "154899b1-075f-4d48-82c6-5c0fb76bff18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:32.652257 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.652155    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/154899b1-075f-4d48-82c6-5c0fb76bff18-istiod-ca-cert" pod "154899b1-075f-4d48-82c6-5c0fb76bff18" (UID: "154899b1-075f-4d48-82c6-5c0fb76bff18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:01:32.652568 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.652367    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-data" pod "154899b1-075f-4d48-82c6-5c0fb76bff18" (UID: "154899b1-075f-4d48-82c6-5c0fb76bff18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:32.652568 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.652511    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-workload-certs" pod "154899b1-075f-4d48-82c6-5c0fb76bff18" (UID: "154899b1-075f-4d48-82c6-5c0fb76bff18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:32.653029 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.653005    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:32.653029 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.653029    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154899b1-075f-4d48-82c6-5c0fb76bff18-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:32.653275 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.653039    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:32.653275 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.653048    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:32.653275 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.653056    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:32.654886 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.654863    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-envoy" pod "154899b1-075f-4d48-82c6-5c0fb76bff18" (UID: "154899b1-075f-4d48-82c6-5c0fb76bff18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:32.654886 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.654878    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-podinfo" pod "154899b1-075f-4d48-82c6-5c0fb76bff18" (UID: "154899b1-075f-4d48-82c6-5c0fb76bff18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:01:32.655033 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.654929    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-token" pod "154899b1-075f-4d48-82c6-5c0fb76bff18" (UID: "154899b1-075f-4d48-82c6-5c0fb76bff18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:01:32.655033 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.654953    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154899b1-075f-4d48-82c6-5c0fb76bff18-kube-api-access-cqcbn" pod "154899b1-075f-4d48-82c6-5c0fb76bff18" (UID: "154899b1-075f-4d48-82c6-5c0fb76bff18"). InnerVolumeSpecName "kube-api-access-cqcbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:01:32.754552 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.754475    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqcbn\" (UniqueName: \"kubernetes.io/projected/154899b1-075f-4d48-82c6-5c0fb76bff18-kube-api-access-cqcbn\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:32.754552 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.754525    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:32.754552 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.754535    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:32.754552 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.754544    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154899b1-075f-4d48-82c6-5c0fb76bff18-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:32.827900 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.827797    5650 generic.go:356] "Generic (PLEG): container finished" podID="154899b1-075f-4d48-82c6-5c0fb76bff18" containerID="c64317c691c04b77ba04327184b72a3f6d2fdd6bfcc2d80fbf3ffefd3d1880f8" exitCode=0
Oct 31 02:01:32.828065 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.827891    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"
Oct 31 02:01:32.828065 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.827907    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph" event={"ID":"154899b1-075f-4d48-82c6-5c0fb76bff18","Type":"ContainerDied","Data":"c64317c691c04b77ba04327184b72a3f6d2fdd6bfcc2d80fbf3ffefd3d1880f8"}
Oct 31 02:01:32.828065 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.827979    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph" event={"ID":"154899b1-075f-4d48-82c6-5c0fb76bff18","Type":"ContainerDied","Data":"7f5ae611f0e4da513c7b52d379ffe15bd525382b762c3f39e9be0897cb24ec3c"}
Oct 31 02:01:32.828065 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.828010    5650 scope.go:122] "RemoveContainer" containerID="c64317c691c04b77ba04327184b72a3f6d2fdd6bfcc2d80fbf3ffefd3d1880f8"
Oct 31 02:01:32.850917 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.850888    5650 scope.go:122] "RemoveContainer" containerID="c64317c691c04b77ba04327184b72a3f6d2fdd6bfcc2d80fbf3ffefd3d1880f8"
Oct 31 02:01:32.851555 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:01:32.851518    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c64317c691c04b77ba04327184b72a3f6d2fdd6bfcc2d80fbf3ffefd3d1880f8\": container with ID starting with c64317c691c04b77ba04327184b72a3f6d2fdd6bfcc2d80fbf3ffefd3d1880f8 not found: ID does not exist" containerID="c64317c691c04b77ba04327184b72a3f6d2fdd6bfcc2d80fbf3ffefd3d1880f8"
Oct 31 02:01:32.851696 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.851563    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c64317c691c04b77ba04327184b72a3f6d2fdd6bfcc2d80fbf3ffefd3d1880f8"} err="failed to get container status \"c64317c691c04b77ba04327184b72a3f6d2fdd6bfcc2d80fbf3ffefd3d1880f8\": rpc error: code = NotFound desc = could not find container \"c64317c691c04b77ba04327184b72a3f6d2fdd6bfcc2d80fbf3ffefd3d1880f8\": container with ID starting with c64317c691c04b77ba04327184b72a3f6d2fdd6bfcc2d80fbf3ffefd3d1880f8 not found: ID does not exist"
Oct 31 02:01:32.861541 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.861439    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"]
Oct 31 02:01:32.869087 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:32.869036    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-v8lph"]
Oct 31 02:01:34.125399 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:34.125357    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="154899b1-075f-4d48-82c6-5c0fb76bff18" path="/var/lib/kubelet/pods/154899b1-075f-4d48-82c6-5c0fb76bff18/volumes"
Oct 31 02:01:38.967476 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:38.963647    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx" podUID="3aa84cad-c52d-4da2-bb1f-622bf9738e23" containerName="main" containerID="cri-o://17ed1c83938cb18306a59f3720c9ad9a8d583a70ffa05eb9d6f6516510c2ea82" gracePeriod=45
Oct 31 02:01:39.024685 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:39.020148    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx" podUID="3aa84cad-c52d-4da2-bb1f-622bf9738e23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.176:8000/ready\": EOF"
Oct 31 02:01:39.870689 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:39.870654    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 02:01:39.933468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:39.931312    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-home\" (UniqueName: \"kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-home\") pod \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\" (UID: \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\") "
Oct 31 02:01:39.933468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:39.931416    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3aa84cad-c52d-4da2-bb1f-622bf9738e23-tls-certs\" (UniqueName: \"kubernetes.io/secret/3aa84cad-c52d-4da2-bb1f-622bf9738e23-tls-certs\") pod \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\" (UID: \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\") "
Oct 31 02:01:39.933468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:39.931476    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa84cad-c52d-4da2-bb1f-622bf9738e23-kube-api-access-p2sqk\" (UniqueName: \"kubernetes.io/projected/3aa84cad-c52d-4da2-bb1f-622bf9738e23-kube-api-access-p2sqk\") pod \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\" (UID: \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\") "
Oct 31 02:01:39.933468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:39.931534    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-dshm\") pod \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\" (UID: \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\") "
Oct 31 02:01:39.933468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:39.931704    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-tmp-dir\") pod \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\" (UID: \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\") "
Oct 31 02:01:39.933468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:39.931737    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-model-cache\") pod \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\" (UID: \"3aa84cad-c52d-4da2-bb1f-622bf9738e23\") "
Oct 31 02:01:39.940468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:39.938110    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-model-cache" pod "3aa84cad-c52d-4da2-bb1f-622bf9738e23" (UID: "3aa84cad-c52d-4da2-bb1f-622bf9738e23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:39.940468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:39.939137    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-home" pod "3aa84cad-c52d-4da2-bb1f-622bf9738e23" (UID: "3aa84cad-c52d-4da2-bb1f-622bf9738e23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:39.940468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:39.939306    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-tmp-dir" pod "3aa84cad-c52d-4da2-bb1f-622bf9738e23" (UID: "3aa84cad-c52d-4da2-bb1f-622bf9738e23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:39.945566 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:39.945080    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aa84cad-c52d-4da2-bb1f-622bf9738e23-tls-certs" pod "3aa84cad-c52d-4da2-bb1f-622bf9738e23" (UID: "3aa84cad-c52d-4da2-bb1f-622bf9738e23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:01:39.949120 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:39.949026    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-dshm" pod "3aa84cad-c52d-4da2-bb1f-622bf9738e23" (UID: "3aa84cad-c52d-4da2-bb1f-622bf9738e23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:01:39.949386 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:39.949359    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa84cad-c52d-4da2-bb1f-622bf9738e23-kube-api-access-p2sqk" pod "3aa84cad-c52d-4da2-bb1f-622bf9738e23" (UID: "3aa84cad-c52d-4da2-bb1f-622bf9738e23"). InnerVolumeSpecName "kube-api-access-p2sqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:01:40.039650 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:40.039525    5650 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-tmp-dir\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:40.039650 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:40.039553    5650 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-model-cache\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:40.039650 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:40.039563    5650 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-home\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:40.039650 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:40.039572    5650 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3aa84cad-c52d-4da2-bb1f-622bf9738e23-tls-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:40.039650 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:40.039580    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2sqk\" (UniqueName: \"kubernetes.io/projected/3aa84cad-c52d-4da2-bb1f-622bf9738e23-kube-api-access-p2sqk\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:40.039650 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:40.039591    5650 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aa84cad-c52d-4da2-bb1f-622bf9738e23-dshm\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:01:40.040417 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:40.040160    5650 generic.go:356] "Generic (PLEG): container finished" podID="3aa84cad-c52d-4da2-bb1f-622bf9738e23" containerID="17ed1c83938cb18306a59f3720c9ad9a8d583a70ffa05eb9d6f6516510c2ea82" exitCode=0
Oct 31 02:01:40.040417 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:40.040325    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx" event={"ID":"3aa84cad-c52d-4da2-bb1f-622bf9738e23","Type":"ContainerDied","Data":"17ed1c83938cb18306a59f3720c9ad9a8d583a70ffa05eb9d6f6516510c2ea82"}
Oct 31 02:01:40.040417 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:40.040345    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"
Oct 31 02:01:40.040417 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:40.040371    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx" event={"ID":"3aa84cad-c52d-4da2-bb1f-622bf9738e23","Type":"ContainerDied","Data":"17dbbead1f52ba250dc549596c206a53f69e418e91f967bb81ef9e973e93390c"}
Oct 31 02:01:40.040417 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:40.040392    5650 scope.go:122] "RemoveContainer" containerID="17ed1c83938cb18306a59f3720c9ad9a8d583a70ffa05eb9d6f6516510c2ea82"
Oct 31 02:01:40.093421 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:40.091817    5650 scope.go:122] "RemoveContainer" containerID="17ed1c83938cb18306a59f3720c9ad9a8d583a70ffa05eb9d6f6516510c2ea82"
Oct 31 02:01:40.108369 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:01:40.108313    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17ed1c83938cb18306a59f3720c9ad9a8d583a70ffa05eb9d6f6516510c2ea82\": container with ID starting with 17ed1c83938cb18306a59f3720c9ad9a8d583a70ffa05eb9d6f6516510c2ea82 not found: ID does not exist" containerID="17ed1c83938cb18306a59f3720c9ad9a8d583a70ffa05eb9d6f6516510c2ea82"
Oct 31 02:01:40.108723 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:40.108384    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ed1c83938cb18306a59f3720c9ad9a8d583a70ffa05eb9d6f6516510c2ea82"} err="failed to get container status \"17ed1c83938cb18306a59f3720c9ad9a8d583a70ffa05eb9d6f6516510c2ea82\": rpc error: code = NotFound desc = could not find container \"17ed1c83938cb18306a59f3720c9ad9a8d583a70ffa05eb9d6f6516510c2ea82\": container with ID starting with 17ed1c83938cb18306a59f3720c9ad9a8d583a70ffa05eb9d6f6516510c2ea82 not found: ID does not exist"
Oct 31 02:01:40.265469 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:40.260229    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"]
Oct 31 02:01:40.265469 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:40.261755    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx"]
Oct 31 02:01:40.848603 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:40.848550    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-7a850e/auth-test-model-7a850e-kserve-6565d6b7df-prgdx" podUID="3aa84cad-c52d-4da2-bb1f-622bf9738e23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.176:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:01:42.200469 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:42.196148    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aa84cad-c52d-4da2-bb1f-622bf9738e23" path="/var/lib/kubelet/pods/3aa84cad-c52d-4da2-bb1f-622bf9738e23/volumes"
Oct 31 02:01:46.377366 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:46.377331    5650 scope.go:122] "RemoveContainer" containerID="db62156159d40902d78d24677dcea511d5d4aa504f4dcaaf60f89b8995029ecf"
Oct 31 02:01:46.429060 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:46.429004    5650 scope.go:122] "RemoveContainer" containerID="3d092f0b7025a6779715bb6434c11b263858fd053e958afc8bce16e6d170564c"
Oct 31 02:01:49.780659 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:49.780601    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 02:01:49.781388 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:49.780693    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg" podUID="cd4d7677-6d53-42e9-8416-024a78dc3ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:01:51.933386 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:51.933350    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"]
Oct 31 02:01:51.934766 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:51.934741    5650 cpu_manager.go:414] "containerMap: removing container" podUID="154899b1-075f-4d48-82c6-5c0fb76bff18" containerName="istio-proxy"
Oct 31 02:01:51.934766 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:51.934765    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154899b1-075f-4d48-82c6-5c0fb76bff18" containerName="istio-proxy"
Oct 31 02:01:51.934925 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:51.934786    5650 cpu_manager.go:414] "containerMap: removing container" podUID="3aa84cad-c52d-4da2-bb1f-622bf9738e23" containerName="main"
Oct 31 02:01:51.934925 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:51.934791    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa84cad-c52d-4da2-bb1f-622bf9738e23" containerName="main"
Oct 31 02:01:51.934925 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:51.934826    5650 cpu_manager.go:414] "containerMap: removing container" podUID="b61e54f0-3295-4e02-aceb-2190f5b00181" containerName="istio-proxy"
Oct 31 02:01:51.934925 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:51.934831    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61e54f0-3295-4e02-aceb-2190f5b00181" containerName="istio-proxy"
Oct 31 02:01:51.934925 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:51.934845    5650 cpu_manager.go:414] "containerMap: removing container" podUID="2d872f42-06a3-4580-8930-fbe87eac3311" containerName="istio-proxy"
Oct 31 02:01:51.934925 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:51.934849    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d872f42-06a3-4580-8930-fbe87eac3311" containerName="istio-proxy"
Oct 31 02:01:51.935135 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:51.934988    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61e54f0-3295-4e02-aceb-2190f5b00181" containerName="istio-proxy"
Oct 31 02:01:51.935135 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:51.934997    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d872f42-06a3-4580-8930-fbe87eac3311" containerName="istio-proxy"
Oct 31 02:01:51.935135 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:51.935008    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="154899b1-075f-4d48-82c6-5c0fb76bff18" containerName="istio-proxy"
Oct 31 02:01:51.935135 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:51.935016    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa84cad-c52d-4da2-bb1f-622bf9738e23" containerName="main"
Oct 31 02:01:52.013192 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.013153    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"]
Oct 31 02:01:52.013364 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.013286    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.135156 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.129918    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.135156 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.130057    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlx8t\" (UniqueName: \"kubernetes.io/projected/f8b0a988-920f-4c28-b813-7ea3c2e108ca-kube-api-access-wlx8t\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.135156 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.130119    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.135156 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.130235    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.135156 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.130263    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.135156 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.130300    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.135156 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.130501    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.135156 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.130546    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.135156 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.130582    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.205619 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.205576    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 02:01:52.205819 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.205645    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc" podUID="c060bf51-21db-4f5b-889d-e655338a9bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:01:52.232028 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.231991    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.232244 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.232054    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlx8t\" (UniqueName: \"kubernetes.io/projected/f8b0a988-920f-4c28-b813-7ea3c2e108ca-kube-api-access-wlx8t\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.232244 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.232079    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.232244 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.232144    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.232244 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.232162    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.232244 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.232195    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.232813 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.232337    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.232813 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.232374    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.232813 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.232403    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.232813 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.232521    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.232813 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.232627    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.232813 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.232762    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.233284 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.232933    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.233284 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.233033    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.323890 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.323850    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.324054 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.323935    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.324141 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.324100    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlx8t\" (UniqueName: \"kubernetes.io/projected/f8b0a988-920f-4c28-b813-7ea3c2e108ca-kube-api-access-wlx8t\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.324180 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.324145    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:52.538018 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:52.537916    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:53.671859 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:53.671814    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn"]
Oct 31 02:01:53.855661 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:53.855377    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn"]
Oct 31 02:01:53.855661 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:53.855538    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn"
Oct 31 02:01:53.984245 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:53.984209    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1b923c0-10b1-4fd5-912f-57d22e29a284-maas-api-tls\") pod \"maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn\" (UID: \"d1b923c0-10b1-4fd5-912f-57d22e29a284\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn"
Oct 31 02:01:53.984441 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:53.984368    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1b923c0-10b1-4fd5-912f-57d22e29a284-metrics-certs\") pod \"maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn\" (UID: \"d1b923c0-10b1-4fd5-912f-57d22e29a284\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn"
Oct 31 02:01:53.984441 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:53.984397    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzbtm\" (UniqueName: \"kubernetes.io/projected/d1b923c0-10b1-4fd5-912f-57d22e29a284-kube-api-access-fzbtm\") pod \"maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn\" (UID: \"d1b923c0-10b1-4fd5-912f-57d22e29a284\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn"
Oct 31 02:01:53.984676 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:53.984651    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1b923c0-10b1-4fd5-912f-57d22e29a284-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn\" (UID: \"d1b923c0-10b1-4fd5-912f-57d22e29a284\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn"
Oct 31 02:01:54.088766 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:54.086982    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1b923c0-10b1-4fd5-912f-57d22e29a284-maas-api-tls\") pod \"maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn\" (UID: \"d1b923c0-10b1-4fd5-912f-57d22e29a284\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn"
Oct 31 02:01:54.088766 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:54.087148    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1b923c0-10b1-4fd5-912f-57d22e29a284-metrics-certs\") pod \"maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn\" (UID: \"d1b923c0-10b1-4fd5-912f-57d22e29a284\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn"
Oct 31 02:01:54.088766 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:54.087190    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzbtm\" (UniqueName: \"kubernetes.io/projected/d1b923c0-10b1-4fd5-912f-57d22e29a284-kube-api-access-fzbtm\") pod \"maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn\" (UID: \"d1b923c0-10b1-4fd5-912f-57d22e29a284\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn"
Oct 31 02:01:54.088766 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:54.087385    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1b923c0-10b1-4fd5-912f-57d22e29a284-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn\" (UID: \"d1b923c0-10b1-4fd5-912f-57d22e29a284\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn"
Oct 31 02:01:54.088766 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:54.088332    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1b923c0-10b1-4fd5-912f-57d22e29a284-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn\" (UID: \"d1b923c0-10b1-4fd5-912f-57d22e29a284\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn"
Oct 31 02:01:54.190519 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:54.190433    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzbtm\" (UniqueName: \"kubernetes.io/projected/d1b923c0-10b1-4fd5-912f-57d22e29a284-kube-api-access-fzbtm\") pod \"maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn\" (UID: \"d1b923c0-10b1-4fd5-912f-57d22e29a284\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn"
Oct 31 02:01:54.219036 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:54.219003    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1b923c0-10b1-4fd5-912f-57d22e29a284-metrics-certs\") pod \"maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn\" (UID: \"d1b923c0-10b1-4fd5-912f-57d22e29a284\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn"
Oct 31 02:01:54.304729 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:54.304697    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-28e4f3-serving-cert\""
Oct 31 02:01:54.307712 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:54.307650    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1b923c0-10b1-4fd5-912f-57d22e29a284-maas-api-tls\") pod \"maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn\" (UID: \"d1b923c0-10b1-4fd5-912f-57d22e29a284\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn"
Oct 31 02:01:54.434260 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:54.433954    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn"
Oct 31 02:01:55.206619 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:55.206565    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 02:01:55.207157 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:55.206649    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m" podUID="2cb82515-c5d2-47a6-af96-d0da5a75bcd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:01:56.535366 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:56.535324    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k"]
Oct 31 02:01:56.794268 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:56.794091    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k"]
Oct 31 02:01:56.795692 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:56.794304    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k"
Oct 31 02:01:56.803838 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:56.803804    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-df9d42-serving-cert\""
Oct 31 02:01:56.885373 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:56.885324    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/149232be-82fa-4fba-8a4f-dba80871843a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k\" (UID: \"149232be-82fa-4fba-8a4f-dba80871843a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k"
Oct 31 02:01:56.885652 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:56.885518    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/149232be-82fa-4fba-8a4f-dba80871843a-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k\" (UID: \"149232be-82fa-4fba-8a4f-dba80871843a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k"
Oct 31 02:01:56.885652 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:56.885565    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2qn4\" (UniqueName: \"kubernetes.io/projected/149232be-82fa-4fba-8a4f-dba80871843a-kube-api-access-f2qn4\") pod \"maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k\" (UID: \"149232be-82fa-4fba-8a4f-dba80871843a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k"
Oct 31 02:01:56.885782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:56.885703    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/149232be-82fa-4fba-8a4f-dba80871843a-metrics-certs\") pod \"maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k\" (UID: \"149232be-82fa-4fba-8a4f-dba80871843a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k"
Oct 31 02:01:56.942517 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:56.942478    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"]
Oct 31 02:01:56.943790 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:01:56.943745    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b0a988_920f_4c28_b813_7ea3c2e108ca.slice/crio-a81211c73516af0a1d299d08130a70d1d9db7acc7c8c69a4e4fd94fdbfae85d4 WatchSource:0}: Error finding container a81211c73516af0a1d299d08130a70d1d9db7acc7c8c69a4e4fd94fdbfae85d4: Status 404 returned error can't find the container with id a81211c73516af0a1d299d08130a70d1d9db7acc7c8c69a4e4fd94fdbfae85d4
Oct 31 02:01:56.948801 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:56.948760    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:01:56.948924 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:56.948858    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:01:56.948924 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:56.948900    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:01:56.988509 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:56.988433    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/149232be-82fa-4fba-8a4f-dba80871843a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k\" (UID: \"149232be-82fa-4fba-8a4f-dba80871843a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k"
Oct 31 02:01:56.988725 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:56.988556    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/149232be-82fa-4fba-8a4f-dba80871843a-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k\" (UID: \"149232be-82fa-4fba-8a4f-dba80871843a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k"
Oct 31 02:01:56.988725 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:56.988585    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2qn4\" (UniqueName: \"kubernetes.io/projected/149232be-82fa-4fba-8a4f-dba80871843a-kube-api-access-f2qn4\") pod \"maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k\" (UID: \"149232be-82fa-4fba-8a4f-dba80871843a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k"
Oct 31 02:01:56.988725 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:56.988676    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/149232be-82fa-4fba-8a4f-dba80871843a-metrics-certs\") pod \"maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k\" (UID: \"149232be-82fa-4fba-8a4f-dba80871843a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k"
Oct 31 02:01:56.989698 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:56.989632    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/149232be-82fa-4fba-8a4f-dba80871843a-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k\" (UID: \"149232be-82fa-4fba-8a4f-dba80871843a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k"
Oct 31 02:01:56.992017 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:56.991921    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/149232be-82fa-4fba-8a4f-dba80871843a-metrics-certs\") pod \"maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k\" (UID: \"149232be-82fa-4fba-8a4f-dba80871843a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k"
Oct 31 02:01:56.992150 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:56.992077    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/149232be-82fa-4fba-8a4f-dba80871843a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k\" (UID: \"149232be-82fa-4fba-8a4f-dba80871843a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k"
Oct 31 02:01:57.005196 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:57.005164    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2qn4\" (UniqueName: \"kubernetes.io/projected/149232be-82fa-4fba-8a4f-dba80871843a-kube-api-access-f2qn4\") pod \"maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k\" (UID: \"149232be-82fa-4fba-8a4f-dba80871843a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k"
Oct 31 02:01:57.030814 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:57.030785    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k"
Oct 31 02:01:57.062851 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:57.062807    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn"]
Oct 31 02:01:57.063204 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:01:57.063178    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b923c0_10b1_4fd5_912f_57d22e29a284.slice/crio-acb07badfc071ccfec7b8b7f771e42c0403cd0dfd7c6541efffc525303fa3807 WatchSource:0}: Error finding container acb07badfc071ccfec7b8b7f771e42c0403cd0dfd7c6541efffc525303fa3807: Status 404 returned error can't find the container with id acb07badfc071ccfec7b8b7f771e42c0403cd0dfd7c6541efffc525303fa3807
Oct 31 02:01:57.612906 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:01:57.612179    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod149232be_82fa_4fba_8a4f_dba80871843a.slice/crio-da56da3b3d9358b6ba8164fd62388e981c7a7b1203c13b6ee0cf852fd2ccec35 WatchSource:0}: Error finding container da56da3b3d9358b6ba8164fd62388e981c7a7b1203c13b6ee0cf852fd2ccec35: Status 404 returned error can't find the container with id da56da3b3d9358b6ba8164fd62388e981c7a7b1203c13b6ee0cf852fd2ccec35
Oct 31 02:01:57.618765 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:57.618731    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k"]
Oct 31 02:01:57.817602 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:57.817538    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn" event={"ID":"d1b923c0-10b1-4fd5-912f-57d22e29a284","Type":"ContainerStarted","Data":"acb07badfc071ccfec7b8b7f771e42c0403cd0dfd7c6541efffc525303fa3807"}
Oct 31 02:01:57.820140 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:57.820066    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k" event={"ID":"149232be-82fa-4fba-8a4f-dba80871843a","Type":"ContainerStarted","Data":"da56da3b3d9358b6ba8164fd62388e981c7a7b1203c13b6ee0cf852fd2ccec35"}
Oct 31 02:01:57.823758 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:57.823399    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9" event={"ID":"f8b0a988-920f-4c28-b813-7ea3c2e108ca","Type":"ContainerStarted","Data":"ab00ada38fd686ef62012a9cd8d625bf2d2a6c45a5c00510cfd496db0fb265d6"}
Oct 31 02:01:57.823758 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:57.823494    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9" event={"ID":"f8b0a988-920f-4c28-b813-7ea3c2e108ca","Type":"ContainerStarted","Data":"a81211c73516af0a1d299d08130a70d1d9db7acc7c8c69a4e4fd94fdbfae85d4"}
Oct 31 02:01:57.853900 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:57.853826    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9" podStartSLOduration=6.853796337 podStartE2EDuration="6.853796337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:57.850900382 +0000 UTC m=+2473.854614677" watchObservedRunningTime="2026-10-31 02:01:57.853796337 +0000 UTC m=+2473.857510629"
Oct 31 02:01:58.383408 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:58.383336    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:01:58.384866 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:58.384671    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 31 02:01:58.384866 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:58.384857    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9" podUID="f8b0a988-920f-4c28-b813-7ea3c2e108ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 31 02:01:58.746961 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:01:58.746916    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:01:58.861563 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:58.861473    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn" event={"ID":"d1b923c0-10b1-4fd5-912f-57d22e29a284","Type":"ContainerStarted","Data":"a94e6c4e3f09a426edeed1aec734bc4ff4abc8608389687263d70af91e73a192"}
Oct 31 02:01:58.863804 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:58.862582    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn"
Oct 31 02:01:58.866095 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:58.866008    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k" event={"ID":"149232be-82fa-4fba-8a4f-dba80871843a","Type":"ContainerStarted","Data":"183feb5a08f976e420d32da3ef7dcfba6d8be0d80d6f99f6d1bfefd754426280"}
Oct 31 02:01:58.866214 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:58.866155    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k"
Oct 31 02:01:58.905471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:58.902425    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn" podStartSLOduration=5.133284576 podStartE2EDuration="5.902390826s" totalImagesPullingTime="769.10625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:58.893021393 +0000 UTC m=+2474.896735685" watchObservedRunningTime="2026-10-31 02:01:58.902390826 +0000 UTC m=+2474.906105118"
Oct 31 02:01:58.952455 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:58.948695    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k" podStartSLOduration=2.265873285 podStartE2EDuration="2.948675287s" totalImagesPullingTime="682.802002ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:58.943887927 +0000 UTC m=+2474.947602220" watchObservedRunningTime="2026-10-31 02:01:58.948675287 +0000 UTC m=+2474.952389579"
Oct 31 02:01:59.199531 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:59.196614    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"]
Oct 31 02:01:59.199531 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:59.197013    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2" podUID="6aa3fc04-120c-4dd7-ba46-99b133d092b0" containerName="istio-proxy" containerID="cri-o://7c77695d20de8042934cc6056a815aa4fdb7fdc9dcf2b6a2a65c0eebfc676bfc" gracePeriod=30
Oct 31 02:01:59.382703 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:59.382666    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 31 02:01:59.382880 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:01:59.382721    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9" podUID="f8b0a988-920f-4c28-b813-7ea3c2e108ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 31 02:02:00.382926 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:00.382894    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 31 02:02:00.383610 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:00.383573    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9" podUID="f8b0a988-920f-4c28-b813-7ea3c2e108ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 31 02:02:01.382049 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:01.382014    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 31 02:02:01.382254 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:01.382069    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9" podUID="f8b0a988-920f-4c28-b813-7ea3c2e108ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 31 02:02:02.383637 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:02.383594    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 31 02:02:02.390518 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:02.390461    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9" podUID="f8b0a988-920f-4c28-b813-7ea3c2e108ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 31 02:02:03.147281 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.147218    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"]
Oct 31 02:02:03.237325 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.203699    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"]
Oct 31 02:02:03.237325 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.203762    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"]
Oct 31 02:02:03.237325 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.236311    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.249910 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.249183    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.254636 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.254564    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"]
Oct 31 02:02:03.268077 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.267782    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.268077 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.267838    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.268077 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.267988    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.268077 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.268065    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e22feaff-a34e-4be5-b480-30b9db5c5d58-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.268553 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.268111    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.268553 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.268211    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.268977 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.268666    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.268977 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.268752    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.268977 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.268802    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.268977 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.268861    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.268977 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.268887    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.268977 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.268944    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.268977 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.268971    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.269542 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.269517    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zk22\" (UniqueName: \"kubernetes.io/projected/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-kube-api-access-5zk22\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.269628 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.269554    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.269628 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.269587    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tbkx\" (UniqueName: \"kubernetes.io/projected/e22feaff-a34e-4be5-b480-30b9db5c5d58-kube-api-access-7tbkx\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.269755 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.269663    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.269755 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.269697    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.371774 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.371731    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.372261 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.372234    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.372563 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.372534    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.372674 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.372653    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.372872 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.372828    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.373091 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.373069    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.373194 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.373110    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.373194 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.373186    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.373315 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.373213    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.373315 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.373271    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.373315 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.373298    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.373535 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.373366    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zk22\" (UniqueName: \"kubernetes.io/projected/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-kube-api-access-5zk22\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.373535 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.373391    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.373535 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.373423    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tbkx\" (UniqueName: \"kubernetes.io/projected/e22feaff-a34e-4be5-b480-30b9db5c5d58-kube-api-access-7tbkx\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.373698 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.373540    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.373698 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.373581    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.373807 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.373713    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.373807 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.373748    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.373948 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.373868    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.374011 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.373946    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e22feaff-a34e-4be5-b480-30b9db5c5d58-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.374011 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.373993    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.374202 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.374171    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.374639 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.374614    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.374744 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.374719    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.375296 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.375240    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.375483 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.375434    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.375561 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.375467    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.375629 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.375578    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e22feaff-a34e-4be5-b480-30b9db5c5d58-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.377484 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.377323    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.377702 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.377659    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.377897 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.377874    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.377968 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.377894    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.382181 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.382149    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 31 02:02:03.382291 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.382211    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9" podUID="f8b0a988-920f-4c28-b813-7ea3c2e108ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 31 02:02:03.389729 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.389695    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.391886 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.391795    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tbkx\" (UniqueName: \"kubernetes.io/projected/e22feaff-a34e-4be5-b480-30b9db5c5d58-kube-api-access-7tbkx\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.393769 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.393742    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zk22\" (UniqueName: \"kubernetes.io/projected/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-kube-api-access-5zk22\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.395344 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.395321    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:03.591996 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.591951    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:03.624100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:03.623936    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:04.252385 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.250734    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:04.252385 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.250806    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:04.252385 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.250835    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:04.252385 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.250989    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"]
Oct 31 02:02:04.289945 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.286880    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"]
Oct 31 02:02:04.309473 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:02:04.291646    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d3adaa_ed6d_46d1_8990_62b11d6d0adc.slice/crio-f4c5b509836bfec59f4740d54718614cdca3800773ee82108e3670eca9f25c5a WatchSource:0}: Error finding container f4c5b509836bfec59f4740d54718614cdca3800773ee82108e3670eca9f25c5a: Status 404 returned error can't find the container with id f4c5b509836bfec59f4740d54718614cdca3800773ee82108e3670eca9f25c5a
Oct 31 02:02:04.323695 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.322826    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:04.323695 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.322922    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:04.323695 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.322963    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:04.397292 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.397261    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:02:04.399737 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.399518    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:02:04.401110 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.401052    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:02:04.781620 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.781580    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 02:02:04.781798 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.781656    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg" podUID="cd4d7677-6d53-42e9-8416-024a78dc3ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:04.823324 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.823280    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 02:02:04.891835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.889661    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k"
Oct 31 02:02:04.956815 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.956748    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-data\") pod \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") "
Oct 31 02:02:04.956996 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.956898    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-podinfo\") pod \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") "
Oct 31 02:02:04.956996 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.956981    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-credential-socket\") pod \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") "
Oct 31 02:02:04.957210 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.957093    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-token\" (UniqueName: \"kubernetes.io/projected/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-token\") pod \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") "
Oct 31 02:02:04.957210 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.957126    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-envoy\") pod \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") "
Oct 31 02:02:04.957210 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.957183    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-workload-socket\") pod \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") "
Oct 31 02:02:04.957706 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.957250    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa3fc04-120c-4dd7-ba46-99b133d092b0-kube-api-access-6b94b\" (UniqueName: \"kubernetes.io/projected/6aa3fc04-120c-4dd7-ba46-99b133d092b0-kube-api-access-6b94b\") pod \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") "
Oct 31 02:02:04.957706 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.957325    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-workload-certs\") pod \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") "
Oct 31 02:02:04.957706 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.957397    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istiod-ca-cert\") pod \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\" (UID: \"6aa3fc04-120c-4dd7-ba46-99b133d092b0\") "
Oct 31 02:02:04.959744 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.959681    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-data" pod "6aa3fc04-120c-4dd7-ba46-99b133d092b0" (UID: "6aa3fc04-120c-4dd7-ba46-99b133d092b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:02:04.959927 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.959893    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-workload-socket" pod "6aa3fc04-120c-4dd7-ba46-99b133d092b0" (UID: "6aa3fc04-120c-4dd7-ba46-99b133d092b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:02:04.960198 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.960135    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-credential-socket" pod "6aa3fc04-120c-4dd7-ba46-99b133d092b0" (UID: "6aa3fc04-120c-4dd7-ba46-99b133d092b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:02:04.960280 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.960198    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-workload-certs" pod "6aa3fc04-120c-4dd7-ba46-99b133d092b0" (UID: "6aa3fc04-120c-4dd7-ba46-99b133d092b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:02:04.960356 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.960337    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istiod-ca-cert" pod "6aa3fc04-120c-4dd7-ba46-99b133d092b0" (UID: "6aa3fc04-120c-4dd7-ba46-99b133d092b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:02:04.984700 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.983956    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"
Oct 31 02:02:04.985151 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.983860    5650 generic.go:356] "Generic (PLEG): container finished" podID="6aa3fc04-120c-4dd7-ba46-99b133d092b0" containerID="7c77695d20de8042934cc6056a815aa4fdb7fdc9dcf2b6a2a65c0eebfc676bfc" exitCode=0
Oct 31 02:02:04.985151 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.984973    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2" event={"ID":"6aa3fc04-120c-4dd7-ba46-99b133d092b0","Type":"ContainerDied","Data":"7c77695d20de8042934cc6056a815aa4fdb7fdc9dcf2b6a2a65c0eebfc676bfc"}
Oct 31 02:02:04.985151 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.985031    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2" event={"ID":"6aa3fc04-120c-4dd7-ba46-99b133d092b0","Type":"ContainerDied","Data":"28e2259b542bf6ad370aa37efaed1103a3a8e5967ff16d3caffa1a5f005aef2b"}
Oct 31 02:02:04.985151 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.985074    5650 scope.go:122] "RemoveContainer" containerID="7c77695d20de8042934cc6056a815aa4fdb7fdc9dcf2b6a2a65c0eebfc676bfc"
Oct 31 02:02:05.000066 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.998914    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9" event={"ID":"e22feaff-a34e-4be5-b480-30b9db5c5d58","Type":"ContainerStarted","Data":"6b87ff1d036353fad9c9c4b880186d559c0f781aa3d0bb4867391d7514b5c801"}
Oct 31 02:02:05.000066 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:04.998980    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9" event={"ID":"e22feaff-a34e-4be5-b480-30b9db5c5d58","Type":"ContainerStarted","Data":"5408a63fb55d0b71db19d8d31ed1f35c46a9ab935031382f858c3f0ed2cea202"}
Oct 31 02:02:05.005889 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.005176    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj" event={"ID":"21d3adaa-ed6d-46d1-8990-62b11d6d0adc","Type":"ContainerStarted","Data":"cb1dd73110454f5319ed9a2801e2d8d1c6603f0813c77b8d4030f88e5c51ecf3"}
Oct 31 02:02:05.005889 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.005235    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj" event={"ID":"21d3adaa-ed6d-46d1-8990-62b11d6d0adc","Type":"ContainerStarted","Data":"f4c5b509836bfec59f4740d54718614cdca3800773ee82108e3670eca9f25c5a"}
Oct 31 02:02:05.034862 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.034799    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa3fc04-120c-4dd7-ba46-99b133d092b0-kube-api-access-6b94b" pod "6aa3fc04-120c-4dd7-ba46-99b133d092b0" (UID: "6aa3fc04-120c-4dd7-ba46-99b133d092b0"). InnerVolumeSpecName "kube-api-access-6b94b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:02:05.036364 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.036299    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-envoy" pod "6aa3fc04-120c-4dd7-ba46-99b133d092b0" (UID: "6aa3fc04-120c-4dd7-ba46-99b133d092b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:02:05.058513 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.055898    5650 scope.go:122] "RemoveContainer" containerID="7c77695d20de8042934cc6056a815aa4fdb7fdc9dcf2b6a2a65c0eebfc676bfc"
Oct 31 02:02:05.061644 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:02:05.058826    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c77695d20de8042934cc6056a815aa4fdb7fdc9dcf2b6a2a65c0eebfc676bfc\": container with ID starting with 7c77695d20de8042934cc6056a815aa4fdb7fdc9dcf2b6a2a65c0eebfc676bfc not found: ID does not exist" containerID="7c77695d20de8042934cc6056a815aa4fdb7fdc9dcf2b6a2a65c0eebfc676bfc"
Oct 31 02:02:05.061644 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.058880    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c77695d20de8042934cc6056a815aa4fdb7fdc9dcf2b6a2a65c0eebfc676bfc"} err="failed to get container status \"7c77695d20de8042934cc6056a815aa4fdb7fdc9dcf2b6a2a65c0eebfc676bfc\": rpc error: code = NotFound desc = could not find container \"7c77695d20de8042934cc6056a815aa4fdb7fdc9dcf2b6a2a65c0eebfc676bfc\": container with ID starting with 7c77695d20de8042934cc6056a815aa4fdb7fdc9dcf2b6a2a65c0eebfc676bfc not found: ID does not exist"
Oct 31 02:02:05.061644 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.060933    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:02:05.061644 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.060960    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:02:05.061644 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.060975    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:02:05.061644 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.060991    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:02:05.061644 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.061006    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b94b\" (UniqueName: \"kubernetes.io/projected/6aa3fc04-120c-4dd7-ba46-99b133d092b0-kube-api-access-6b94b\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:02:05.061644 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.061021    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa3fc04-120c-4dd7-ba46-99b133d092b0-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:02:05.061644 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.061034    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:02:05.105510 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.081360    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj" podStartSLOduration=2.081337009 podStartE2EDuration="2.081337009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:05.077860002 +0000 UTC m=+2481.081574294" watchObservedRunningTime="2026-10-31 02:02:05.081337009 +0000 UTC m=+2481.085051302"
Oct 31 02:02:05.105510 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.084740    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-token" pod "6aa3fc04-120c-4dd7-ba46-99b133d092b0" (UID: "6aa3fc04-120c-4dd7-ba46-99b133d092b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:02:05.121498 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.120160    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-podinfo" pod "6aa3fc04-120c-4dd7-ba46-99b133d092b0" (UID: "6aa3fc04-120c-4dd7-ba46-99b133d092b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:02:05.139536 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.138724    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9" podStartSLOduration=2.138700992 podStartE2EDuration="2.138700992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:05.033934084 +0000 UTC m=+2481.037648376" watchObservedRunningTime="2026-10-31 02:02:05.138700992 +0000 UTC m=+2481.142415285"
Oct 31 02:02:05.165263 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.165228    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:02:05.165263 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.165259    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa3fc04-120c-4dd7-ba46-99b133d092b0-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:02:05.321971 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.321920    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"]
Oct 31 02:02:05.329978 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.329939    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-8dwl2"]
Oct 31 02:02:05.594482 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.594380    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:05.595948 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.595917    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Oct 31 02:02:05.596080 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.595981    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj" podUID="21d3adaa-ed6d-46d1-8990-62b11d6d0adc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Oct 31 02:02:05.627228 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.626859    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:05.631556 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.629913    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 31 02:02:05.631556 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.629979    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9" podUID="e22feaff-a34e-4be5-b480-30b9db5c5d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 31 02:02:05.900745 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:05.899240    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn"
Oct 31 02:02:06.166735 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:06.166591    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aa3fc04-120c-4dd7-ba46-99b133d092b0" path="/var/lib/kubelet/pods/6aa3fc04-120c-4dd7-ba46-99b133d092b0/volumes"
Oct 31 02:02:06.593217 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:06.593183    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Oct 31 02:02:06.593398 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:06.593242    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj" podUID="21d3adaa-ed6d-46d1-8990-62b11d6d0adc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Oct 31 02:02:06.627093 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:06.626636    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 31 02:02:06.627093 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:06.626713    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9" podUID="e22feaff-a34e-4be5-b480-30b9db5c5d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 31 02:02:06.985089 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:06.984489    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"]
Oct 31 02:02:06.990051 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:06.989295    5650 cpu_manager.go:414] "containerMap: removing container" podUID="6aa3fc04-120c-4dd7-ba46-99b133d092b0" containerName="istio-proxy"
Oct 31 02:02:06.990051 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:06.989329    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa3fc04-120c-4dd7-ba46-99b133d092b0" containerName="istio-proxy"
Oct 31 02:02:07.062108 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.033512    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa3fc04-120c-4dd7-ba46-99b133d092b0" containerName="istio-proxy"
Oct 31 02:02:07.089014 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.088643    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"]
Oct 31 02:02:07.089014 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.088701    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"]
Oct 31 02:02:07.090787 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.089396    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.164494 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.158879    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"]
Oct 31 02:02:07.164494 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.159026    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.196592 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.196254    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.196592 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.196309    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czh9g\" (UniqueName: \"kubernetes.io/projected/cbf0e856-3662-486c-94d1-c13cc57b633a-kube-api-access-czh9g\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.223471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.196439    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.223471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.197325    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.223471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.197404    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.223471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.197484    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf0e856-3662-486c-94d1-c13cc57b633a-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.223471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.197684    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.223471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.197716    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.223471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.197788    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.251539 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.237552    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 02:02:07.251539 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.237644    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc" podUID="c060bf51-21db-4f5b-889d-e655338a9bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:07.299416 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.299377    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.323470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.299903    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.323470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.322607    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf0e856-3662-486c-94d1-c13cc57b633a-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.323470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.322757    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.323470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.322818    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.323470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.322979    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.323470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.323046    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.323470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.323097    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.323470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.323192    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.323470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.323287    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.323470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.323371    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa28245-e7ac-49b8-a507-4b452bb944e7-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.324223 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.323520    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.324223 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.323569    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czh9g\" (UniqueName: \"kubernetes.io/projected/cbf0e856-3662-486c-94d1-c13cc57b633a-kube-api-access-czh9g\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.324223 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.323642    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.324223 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.323737    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wth6\" (UniqueName: \"kubernetes.io/projected/bfa28245-e7ac-49b8-a507-4b452bb944e7-kube-api-access-2wth6\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.324223 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.323847    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.324223 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.323879    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.324223 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.323947    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.324223 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.323980    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.346994 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.326899    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.346994 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.327686    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf0e856-3662-486c-94d1-c13cc57b633a-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.346994 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.327982    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.346994 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.335589    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.376019 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.375981    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.393028 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.390796    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czh9g\" (UniqueName: \"kubernetes.io/projected/cbf0e856-3662-486c-94d1-c13cc57b633a-kube-api-access-czh9g\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.393028 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.390900    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.395034 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.394952    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.435118 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.435071    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.438471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.435370    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.449528 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.440201    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.449528 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.440541    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.450094 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.450036    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.450355 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.450335    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.450681 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.450658    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa28245-e7ac-49b8-a507-4b452bb944e7-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.451021 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.450989    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.451124 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.451085    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wth6\" (UniqueName: \"kubernetes.io/projected/bfa28245-e7ac-49b8-a507-4b452bb944e7-kube-api-access-2wth6\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.451196 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.451178    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.457626 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.451208    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.466439 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.466394    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa28245-e7ac-49b8-a507-4b452bb944e7-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.466774 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.450718    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.467218 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.467191    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.471785 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.471742    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.475410 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.475344    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.486805 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.486725    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.492069 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.491998    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wth6\" (UniqueName: \"kubernetes.io/projected/bfa28245-e7ac-49b8-a507-4b452bb944e7-kube-api-access-2wth6\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:07.587299 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.586961    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:07.593380 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.592968    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Oct 31 02:02:07.593380 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.593042    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj" podUID="21d3adaa-ed6d-46d1-8990-62b11d6d0adc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Oct 31 02:02:07.625472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.625068    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 31 02:02:07.625472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.625151    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9" podUID="e22feaff-a34e-4be5-b480-30b9db5c5d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 31 02:02:07.677405 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:07.677375    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:08.067865 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:08.067826    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"]
Oct 31 02:02:08.096500 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:02:08.096390    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf0e856_3662_486c_94d1_c13cc57b633a.slice/crio-ca1cfd160294ce030352f95cc0471ab504980e46fe2633c199a024c4ab37d930 WatchSource:0}: Error finding container ca1cfd160294ce030352f95cc0471ab504980e46fe2633c199a024c4ab37d930: Status 404 returned error can't find the container with id ca1cfd160294ce030352f95cc0471ab504980e46fe2633c199a024c4ab37d930
Oct 31 02:02:08.140772 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:08.100922    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:08.140772 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:08.101025    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:08.140772 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:08.101078    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:08.289152 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:08.289062    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw" event={"ID":"cbf0e856-3662-486c-94d1-c13cc57b633a","Type":"ContainerStarted","Data":"ca1cfd160294ce030352f95cc0471ab504980e46fe2633c199a024c4ab37d930"}
Oct 31 02:02:08.289152 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:08.289145    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"]
Oct 31 02:02:08.296256 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:08.296216    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:08.296382 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:08.296297    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:08.296382 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:08.296329    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:08.596847 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:08.592413    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Oct 31 02:02:08.596847 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:08.592492    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj" podUID="21d3adaa-ed6d-46d1-8990-62b11d6d0adc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Oct 31 02:02:09.348691 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:08.663866    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 31 02:02:09.348691 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:08.663924    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9" podUID="e22feaff-a34e-4be5-b480-30b9db5c5d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 31 02:02:09.348691 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:09.309082    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": read tcp 10.134.0.2:34630->10.134.0.206:15021: read: connection reset by peer" start-of-body=
Oct 31 02:02:09.348691 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:09.309149    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m" podUID="2cb82515-c5d2-47a6-af96-d0da5a75bcd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": read tcp 10.134.0.2:34630->10.134.0.206:15021: read: connection reset by peer"
Oct 31 02:02:09.348691 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:09.324288    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m" event={"ID":"bfa28245-e7ac-49b8-a507-4b452bb944e7","Type":"ContainerStarted","Data":"9a85ecca2d62a9abe6dbf01a8437fbebb8d3643bd1fd44f3b0ddca3adc05e2a4"}
Oct 31 02:02:09.348691 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:09.324357    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m" event={"ID":"bfa28245-e7ac-49b8-a507-4b452bb944e7","Type":"ContainerStarted","Data":"73966d8edb6c33aad4d35e192ee3a42c97bd3cb7429f2e09d43450e622a667bc"}
Oct 31 02:02:09.348691 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:09.340020    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw" event={"ID":"cbf0e856-3662-486c-94d1-c13cc57b633a","Type":"ContainerStarted","Data":"518442071a6687e615ccc4b3c0da2c8852311c1918f83b9bb2277ca222cc8cb4"}
Oct 31 02:02:09.388143 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:09.387134    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m" podStartSLOduration=3.387111613 podStartE2EDuration="3.387111613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:09.384770768 +0000 UTC m=+2485.388485096" watchObservedRunningTime="2026-10-31 02:02:09.387111613 +0000 UTC m=+2485.390825905"
Oct 31 02:02:09.592389 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:09.591517    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:09.593583 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:09.593171    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 31 02:02:09.593583 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:09.593216    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw" podUID="cbf0e856-3662-486c-94d1-c13cc57b633a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 31 02:02:09.593583 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:09.593429    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Oct 31 02:02:09.593583 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:09.593468    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj" podUID="21d3adaa-ed6d-46d1-8990-62b11d6d0adc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Oct 31 02:02:09.626194 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:09.625282    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 31 02:02:09.626194 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:09.625376    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9" podUID="e22feaff-a34e-4be5-b480-30b9db5c5d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 31 02:02:09.698467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:09.680314    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:09.698467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:09.680477    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 31 02:02:09.698467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:09.680532    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m" podUID="bfa28245-e7ac-49b8-a507-4b452bb944e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 31 02:02:10.495359 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:10.490664    5650 generic.go:356] "Generic (PLEG): container finished" podID="2cb82515-c5d2-47a6-af96-d0da5a75bcd3" containerID="722b81cc31875b9ef514423be1993eaa0c5106f7b32341981e17336b1ec009f8" exitCode=0
Oct 31 02:02:10.497016 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:10.496979    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m" event={"ID":"2cb82515-c5d2-47a6-af96-d0da5a75bcd3","Type":"ContainerDied","Data":"722b81cc31875b9ef514423be1993eaa0c5106f7b32341981e17336b1ec009f8"}
Oct 31 02:02:10.523473 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:10.511381    5650 scope.go:122] "RemoveContainer" containerID="722b81cc31875b9ef514423be1993eaa0c5106f7b32341981e17336b1ec009f8"
Oct 31 02:02:10.548180 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:10.547656    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:10.548180 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:10.547771    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:10.548180 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:10.547823    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:10.594922 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:10.594391    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 31 02:02:10.594922 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:10.594470    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw" podUID="cbf0e856-3662-486c-94d1-c13cc57b633a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 31 02:02:10.594922 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:10.594821    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Oct 31 02:02:10.594922 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:10.594855    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj" podUID="21d3adaa-ed6d-46d1-8990-62b11d6d0adc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Oct 31 02:02:10.705479 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:10.684562    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw" podStartSLOduration=4.684535794 podStartE2EDuration="4.684535794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:09.453234009 +0000 UTC m=+2485.456948301" watchObservedRunningTime="2026-10-31 02:02:10.684535794 +0000 UTC m=+2486.688250087"
Oct 31 02:02:10.771773 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:10.748549    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 31 02:02:10.771773 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:10.748620    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9" podUID="e22feaff-a34e-4be5-b480-30b9db5c5d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 31 02:02:10.780185 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:10.778134    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 31 02:02:10.780185 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:10.778197    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m" podUID="bfa28245-e7ac-49b8-a507-4b452bb944e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 31 02:02:10.989472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:10.938567    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:11.427947 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:11.417300    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 02:02:11.511535 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:11.496673    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:11.600594 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:11.596854    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 02:02:11.655595 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:11.630201    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:11.668508 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:11.597709    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 31 02:02:11.668508 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:11.655929    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw" podUID="cbf0e856-3662-486c-94d1-c13cc57b633a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 31 02:02:11.668508 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:11.635686    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Oct 31 02:02:11.668508 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:11.656009    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj" podUID="21d3adaa-ed6d-46d1-8990-62b11d6d0adc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Oct 31 02:02:11.668508 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:11.656391    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 31 02:02:11.668508 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:11.656473    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9" podUID="e22feaff-a34e-4be5-b480-30b9db5c5d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 31 02:02:11.668508 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:11.656784    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:11.716396 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:11.716324    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:11.758485 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:11.748250    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 31 02:02:11.758485 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:11.748688    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m" podUID="bfa28245-e7ac-49b8-a507-4b452bb944e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 31 02:02:12.002528 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:11.961614    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:12.002528 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:11.962133    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 02:02:12.273528 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:12.273328    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded"
Oct 31 02:02:12.417673 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:12.417479    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:12.497263 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:12.497094    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:12.719411 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:12.705523    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:12.719411 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:12.714153    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 02:02:12.719411 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:12.717462    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 31 02:02:12.719411 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:12.717513    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m" podUID="bfa28245-e7ac-49b8-a507-4b452bb944e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 31 02:02:12.783555 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:12.758252    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded"
Oct 31 02:02:12.812477 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:12.806543    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 31 02:02:12.812477 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:12.806607    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw" podUID="cbf0e856-3662-486c-94d1-c13cc57b633a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 31 02:02:12.812477 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:12.806869    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 02:02:12.833491 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:12.827913    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:12.905555 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:12.902295    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:12.905555 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:12.902406    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 02:02:12.905555 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:12.902463    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 02:02:12.905555 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:12.902497    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 02:02:12.905555 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:12.902534    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 02:02:12.905555 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:12.902568    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 02:02:12.946459 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:12.933391    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 02:02:12.948759 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:12.948695    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:12.985111 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:12.970753    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 02:02:13.128922 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:13.098924    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:13.128922 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:13.114592    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 02:02:13.128922 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:13.114670    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 02:02:13.128922 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:13.114690    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:13.128922 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:13.115876    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:02:13.176162 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:13.155945    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:02:13.176162 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:13.159959    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 02:02:13.176162 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:13.170911    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 02:02:13.179931 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:13.179903    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 02:02:13.198847 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:13.183599    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 02:02:13.280756 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:13.280554    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 02:02:13.597074 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:13.596979    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 31 02:02:13.597363 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:13.597314    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw" podUID="cbf0e856-3662-486c-94d1-c13cc57b633a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 31 02:02:13.679784 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:13.655788    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m" event={"ID":"2cb82515-c5d2-47a6-af96-d0da5a75bcd3","Type":"ContainerStarted","Data":"2217942e8f57fd2623a321bd7bf5055a7ec4310bbd3a1aecf1a2f5d687f7c0ab"}
Oct 31 02:02:13.971063 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:13.970882    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 31 02:02:13.971063 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:13.970941    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m" podUID="bfa28245-e7ac-49b8-a507-4b452bb944e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 31 02:02:14.205494 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:14.090627    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:02:14.205494 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:14.200784    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 31 02:02:14.205494 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:14.200860    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m" podUID="2cb82515-c5d2-47a6-af96-d0da5a75bcd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 31 02:02:14.653756 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:14.642682    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 31 02:02:14.653756 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:14.642761    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw" podUID="cbf0e856-3662-486c-94d1-c13cc57b633a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 31 02:02:14.691477 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:14.679292    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 31 02:02:14.695554 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:14.691811    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m" podUID="bfa28245-e7ac-49b8-a507-4b452bb944e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 31 02:02:15.024780 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:15.004833    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 02:02:15.039768 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:15.030690    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:15.064475 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:15.063594    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 31 02:02:15.064475 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:15.063659    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m" podUID="2cb82515-c5d2-47a6-af96-d0da5a75bcd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 31 02:02:15.127110 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:15.125533    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:15.162366 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:15.156586    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:15.397027 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:15.396410    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:15.397227 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:15.396683    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 31 02:02:15.397671 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:15.396890    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:15.627493 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:15.621616    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 31 02:02:15.627493 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:15.621702    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw" podUID="cbf0e856-3662-486c-94d1-c13cc57b633a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 31 02:02:15.687762 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:15.684098    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 31 02:02:15.688065 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:15.688012    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m" podUID="bfa28245-e7ac-49b8-a507-4b452bb944e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 31 02:02:15.986059 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:15.986023    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"]
Oct 31 02:02:16.055427 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.053833    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 31 02:02:16.055427 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.053899    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m" podUID="2cb82515-c5d2-47a6-af96-d0da5a75bcd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 31 02:02:16.058337 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.058305    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"]
Oct 31 02:02:16.058763 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.058681    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.320472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.284335    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-istio-data\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.320472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.284399    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsl7j\" (UniqueName: \"kubernetes.io/projected/b255f882-402f-4db2-900b-051813a4300a-kube-api-access-lsl7j\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.320472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.284429    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-workload-socket\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.320472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.284476    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b255f882-402f-4db2-900b-051813a4300a-istiod-ca-cert\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.320472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.284513    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-workload-certs\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.320472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.284613    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-istio-envoy\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.320472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.284666    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-credential-socket\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.320472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.284780    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b255f882-402f-4db2-900b-051813a4300a-istio-token\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.320472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.284846    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b255f882-402f-4db2-900b-051813a4300a-istio-podinfo\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.393582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.391061    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-istio-data\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.393582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.391127    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsl7j\" (UniqueName: \"kubernetes.io/projected/b255f882-402f-4db2-900b-051813a4300a-kube-api-access-lsl7j\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.393582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.391161    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-workload-socket\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.393582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.391195    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b255f882-402f-4db2-900b-051813a4300a-istiod-ca-cert\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.393582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.391237    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-workload-certs\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.393582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.391349    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-istio-envoy\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.393582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.391402    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-credential-socket\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.393582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.391575    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b255f882-402f-4db2-900b-051813a4300a-istio-token\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.393582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.391657    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b255f882-402f-4db2-900b-051813a4300a-istio-podinfo\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.393582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.393248    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b255f882-402f-4db2-900b-051813a4300a-istiod-ca-cert\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.395520 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.394411    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-istio-data\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.395520 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.395144    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-workload-socket\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.414368 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.400475    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-credential-socket\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.414766 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.402157    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-workload-certs\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.474152 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.474021    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b255f882-402f-4db2-900b-051813a4300a-istio-token\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.482245 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.482160    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b255f882-402f-4db2-900b-051813a4300a-istio-podinfo\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.489820 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.489742    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-istio-envoy\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.499266 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.499216    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsl7j\" (UniqueName: \"kubernetes.io/projected/b255f882-402f-4db2-900b-051813a4300a-kube-api-access-lsl7j\") pod \"e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") " pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:16.595940 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.594161    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 31 02:02:16.595940 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.594226    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw" podUID="cbf0e856-3662-486c-94d1-c13cc57b633a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 31 02:02:16.678109 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.678061    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 31 02:02:16.678325 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.678127    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m" podUID="bfa28245-e7ac-49b8-a507-4b452bb944e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 31 02:02:16.706868 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:16.706825    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:17.107913 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:17.107869    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 31 02:02:17.108620 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:17.108582    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m" podUID="2cb82515-c5d2-47a6-af96-d0da5a75bcd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 31 02:02:17.410567 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:17.404856    5650 generic.go:356] "Generic (PLEG): container finished" podID="c060bf51-21db-4f5b-889d-e655338a9bfa" containerID="749f9e14419765c55ba413d5e5d93be4f909775c40d3ecb3b30a3d533f6b4c57" exitCode=0
Oct 31 02:02:17.410567 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:17.404926    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc" event={"ID":"c060bf51-21db-4f5b-889d-e655338a9bfa","Type":"ContainerDied","Data":"749f9e14419765c55ba413d5e5d93be4f909775c40d3ecb3b30a3d533f6b4c57"}
Oct 31 02:02:17.410567 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:17.405705    5650 scope.go:122] "RemoveContainer" containerID="749f9e14419765c55ba413d5e5d93be4f909775c40d3ecb3b30a3d533f6b4c57"
Oct 31 02:02:17.433524 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:17.428254    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:17.433524 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:17.428391    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:17.433524 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:17.428489    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:17.622774 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:17.622721    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 31 02:02:17.623190 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:17.623137    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw" podUID="cbf0e856-3662-486c-94d1-c13cc57b633a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 31 02:02:17.906376 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:17.906334    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:17.916474 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:17.915721    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:17.965838 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:17.936716    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:02:18.185028 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.184924    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 31 02:02:18.185703 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.185663    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m" podUID="2cb82515-c5d2-47a6-af96-d0da5a75bcd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 31 02:02:18.187710 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.187315    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:18.257572 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.246566    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:18.257572 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.247108    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 31 02:02:18.576523 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.564721    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"]
Oct 31 02:02:18.647061 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.636825    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"]
Oct 31 02:02:18.647061 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.637930    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.653521 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.649247    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:18.653521 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.650424    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:18.653521 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.650928    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-c088fd-openshift-default-dockercfg-tdwrg\""
Oct 31 02:02:18.654722 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.654246    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:02:18.753765 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.752830    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-workload-certs\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.753765 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.753029    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcc6t\" (UniqueName: \"kubernetes.io/projected/86d5dc42-ef1e-4ce5-9f29-386030d4524b-kube-api-access-jcc6t\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.754762 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.754721    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-credential-socket\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.756399 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.756369    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-envoy\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.756764 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.756734    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-podinfo\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.756870 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.756775    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-workload-socket\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.757214 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.757061    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-data\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.757214 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.757152    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istiod-ca-cert\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.757214 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.757185    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-token\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.849471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.849348    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:18.849873 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.849795    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:18.850114 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.850042    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:18.862120 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.862085    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istiod-ca-cert\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.862309 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.862288    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-token\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.862475 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.862439    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-workload-certs\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.862639 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.862622    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcc6t\" (UniqueName: \"kubernetes.io/projected/86d5dc42-ef1e-4ce5-9f29-386030d4524b-kube-api-access-jcc6t\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.862896 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.862878    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-credential-socket\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.863048 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.863033    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-envoy\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.863265 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.863247    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-podinfo\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.863416 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.863397    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-workload-socket\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.863748 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.863727    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-data\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.864026 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.864002    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istiod-ca-cert\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.864489 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.864468    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-workload-socket\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.864873 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.864854    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-data\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.864995 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.863726    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-workload-certs\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.865406 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.865385    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-credential-socket\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.868393 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.868364    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"]
Oct 31 02:02:18.869884 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.869861    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-envoy\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.870830 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.870346    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-podinfo\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.882545 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.881243    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcc6t\" (UniqueName: \"kubernetes.io/projected/86d5dc42-ef1e-4ce5-9f29-386030d4524b-kube-api-access-jcc6t\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.954317 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.954241    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-token\") pod \"e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") " pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:18.970497 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:18.970250    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:19.053597 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:19.053559    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 31 02:02:19.053796 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:19.053624    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m" podUID="2cb82515-c5d2-47a6-af96-d0da5a75bcd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 31 02:02:19.847516 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:19.821612    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98" event={"ID":"b255f882-402f-4db2-900b-051813a4300a","Type":"ContainerStarted","Data":"6c36837539e4734e79cd5d6a380cf496e212645a8475ef34a23acdb876da0058"}
Oct 31 02:02:19.847516 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:19.821693    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98" event={"ID":"b255f882-402f-4db2-900b-051813a4300a","Type":"ContainerStarted","Data":"5de09eab1b3dd91ad5f86a6995951e1bf5fb680e000c99d232ccea31f207ed81"}
Oct 31 02:02:19.847516 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:19.832931    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 02:02:19.847516 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:19.833035    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg" podUID="cd4d7677-6d53-42e9-8416-024a78dc3ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:20.115738 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:20.115645    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 31 02:02:20.124667 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:20.123571    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m" podUID="2cb82515-c5d2-47a6-af96-d0da5a75bcd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 31 02:02:20.184549 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:20.178961    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc" event={"ID":"c060bf51-21db-4f5b-889d-e655338a9bfa","Type":"ContainerStarted","Data":"522d6b8bb091e65c70faf765b617358495bbca78814bded32b97b96c2c336271"}
Oct 31 02:02:20.184549 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:20.179143    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:20.201805 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:20.201738    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:20.210633 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:20.210080    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:20.242438 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:20.217968    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 02:02:20.242438 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:20.219596    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 02:02:20.264468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:20.248838    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98" podStartSLOduration=5.248816169 podStartE2EDuration="5.248816169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:19.898887886 +0000 UTC m=+2495.902602177" watchObservedRunningTime="2026-10-31 02:02:20.248816169 +0000 UTC m=+2496.252530461"
Oct 31 02:02:20.271821 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:20.264882    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:20.409586 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:20.404375    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"]
Oct 31 02:02:20.540506 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:20.536724    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:20.540506 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:20.536835    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:20.540506 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:20.536901    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:20.591852 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:20.586597    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:20.646200 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:20.645524    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:02:20.676984 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:20.676804    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 31 02:02:20.676984 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:20.676863    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc" podUID="c060bf51-21db-4f5b-889d-e655338a9bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 31 02:02:20.724285 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:20.724013    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:20.724285 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:20.724157    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 31 02:02:20.724285 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:20.724201    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98" podUID="b255f882-402f-4db2-900b-051813a4300a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 31 02:02:20.774995 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:20.774573    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:21.059804 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:21.056292    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 31 02:02:21.059804 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:21.056350    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m" podUID="2cb82515-c5d2-47a6-af96-d0da5a75bcd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 31 02:02:21.163775 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:21.163687    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:02:21.315018 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:21.313893    5650 generic.go:356] "Generic (PLEG): container finished" podID="cd4d7677-6d53-42e9-8416-024a78dc3ff7" containerID="59dc66fa26cf1f535189f4243a938a8ae24805dfaa3854719c2693b9ee087812" exitCode=0
Oct 31 02:02:21.315018 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:21.314146    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg" event={"ID":"cd4d7677-6d53-42e9-8416-024a78dc3ff7","Type":"ContainerDied","Data":"59dc66fa26cf1f535189f4243a938a8ae24805dfaa3854719c2693b9ee087812"}
Oct 31 02:02:21.315018 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:21.314769    5650 scope.go:122] "RemoveContainer" containerID="59dc66fa26cf1f535189f4243a938a8ae24805dfaa3854719c2693b9ee087812"
Oct 31 02:02:21.317990 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:21.317950    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:21.324914 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:21.324880    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:21.325220 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:21.325191    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:21.325867 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:21.321991    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq" event={"ID":"86d5dc42-ef1e-4ce5-9f29-386030d4524b","Type":"ContainerStarted","Data":"d69c8702e8c36ddd6f966c46dba4fd406fb45062605ef40662fe380aac96d8b1"}
Oct 31 02:02:21.655609 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:21.655573    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 31 02:02:21.655875 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:21.655845    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc" podUID="c060bf51-21db-4f5b-889d-e655338a9bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 31 02:02:21.727985 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:21.727947    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 31 02:02:21.728268 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:21.728238    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98" podUID="b255f882-402f-4db2-900b-051813a4300a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 31 02:02:22.226528 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:22.222556    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:02:22.226528 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:22.225852    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:02:22.247862 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:22.247826    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:02:22.665637 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:22.663845    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 31 02:02:22.665637 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:22.663922    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc" podUID="c060bf51-21db-4f5b-889d-e655338a9bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 31 02:02:22.720501 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:22.718938    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 31 02:02:22.720501 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:22.719009    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98" podUID="b255f882-402f-4db2-900b-051813a4300a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 31 02:02:22.739264 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:22.739205    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq" event={"ID":"86d5dc42-ef1e-4ce5-9f29-386030d4524b","Type":"ContainerStarted","Data":"f3bfe1798fd1f9c7b6f9cc2d942897848f4e6ff1772c910fada3fced06f53a6e"}
Oct 31 02:02:22.784799 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:22.784744    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg" event={"ID":"cd4d7677-6d53-42e9-8416-024a78dc3ff7","Type":"ContainerStarted","Data":"6e5dbee1ee1c45eeec16a37cbd6b37e6d23d34d2028ec5d2dead95bf961e9031"}
Oct 31 02:02:22.867846 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:22.867765    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq" podStartSLOduration=5.867742487 podStartE2EDuration="5.867742487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:22.815071966 +0000 UTC m=+2498.818786257" watchObservedRunningTime="2026-10-31 02:02:22.867742487 +0000 UTC m=+2498.871456781"
Oct 31 02:02:22.979564 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:22.956820    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:02:22.979564 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:22.973981    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:23.030488 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:23.028870    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:23.638839 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:23.638799    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 31 02:02:23.639488 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:23.639433    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc" podUID="c060bf51-21db-4f5b-889d-e655338a9bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 31 02:02:23.736587 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:23.736547    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 31 02:02:23.736891 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:23.736863    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98" podUID="b255f882-402f-4db2-900b-051813a4300a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 31 02:02:23.891570 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:23.889118    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:23.893798 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:23.892780    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:02:24.224530 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:24.209692    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 31 02:02:24.224530 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:24.209757    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg" podUID="cd4d7677-6d53-42e9-8416-024a78dc3ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 31 02:02:24.661067 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:24.660972    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 31 02:02:24.680700 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:24.680648    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc" podUID="c060bf51-21db-4f5b-889d-e655338a9bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 31 02:02:24.747789 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:24.747743    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 31 02:02:24.748107 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:24.748057    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98" podUID="b255f882-402f-4db2-900b-051813a4300a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 31 02:02:24.952575 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:24.950002    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 31 02:02:24.952575 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:24.950077    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg" podUID="cd4d7677-6d53-42e9-8416-024a78dc3ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 31 02:02:25.143144 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:25.143098    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:25.190199 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:25.177545    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:25.219280 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:25.219168    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:25.240182 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:25.240138    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:25.262511 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:25.240430    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:25.604537 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:25.603888    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:25.638473 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:25.637655    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 31 02:02:25.638473 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:25.637722    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc" podUID="c060bf51-21db-4f5b-889d-e655338a9bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 31 02:02:25.666105 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:25.666055    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 02:02:25.742470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:25.726679    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 31 02:02:25.742470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:25.726748    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98" podUID="b255f882-402f-4db2-900b-051813a4300a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 31 02:02:25.965480 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:25.958728    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 31 02:02:25.965480 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:25.958795    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg" podUID="cd4d7677-6d53-42e9-8416-024a78dc3ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 31 02:02:25.991250 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:25.991197    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:26.278532 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.256631    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 31 02:02:26.296411 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.279528    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 02:02:26.296996 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.279605    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:26.297274 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.280821    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:26.297848 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.286304    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:26.298508 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.298481    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 02:02:26.592434 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.592339    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 02:02:26.592783 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.592764    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 02:02:26.592961 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.592950    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 02:02:26.593310 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.593281    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 02:02:26.593734 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.593502    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 02:02:26.593907 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.593892    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 02:02:26.597425 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.597401    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 02:02:26.599289 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.599173    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 02:02:26.624471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.607327    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:26.624471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.607915    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 02:02:26.743469 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.718944    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 02:02:26.770510 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.767313    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 02:02:26.772190 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.772158    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 31 02:02:26.798777 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.793555    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded"
Oct 31 02:02:26.798964 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.798814    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98" podUID="b255f882-402f-4db2-900b-051813a4300a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 31 02:02:26.987538 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.960157    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 31 02:02:26.987538 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.960215    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc" podUID="c060bf51-21db-4f5b-889d-e655338a9bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 31 02:02:26.987538 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.962265    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 02:02:26.987538 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.962641    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 31 02:02:26.987538 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.962689    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg" podUID="cd4d7677-6d53-42e9-8416-024a78dc3ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 31 02:02:26.991758 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.991712    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 02:02:26.998786 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:26.998622    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 02:02:27.325714 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:27.325606    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 02:02:28.008471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:28.007753    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:28.159472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:28.152587    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 31 02:02:28.182162 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:28.173085    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc" podUID="c060bf51-21db-4f5b-889d-e655338a9bfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 31 02:02:28.396533 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:28.194963    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:28.397073 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:28.338550    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:02:28.417477 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:28.384042    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 02:02:28.469014 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:28.388309    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 31 02:02:28.469226 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:28.469033    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98" podUID="b255f882-402f-4db2-900b-051813a4300a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 31 02:02:28.469226 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:28.401168    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 31 02:02:28.469226 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:28.469102    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg" podUID="cd4d7677-6d53-42e9-8416-024a78dc3ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 31 02:02:28.470117 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:28.452544    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 02:02:28.693468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:28.693367    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg"]
Oct 31 02:02:28.964599 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:28.880773    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:02:28.964599 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:28.894747    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg"
Oct 31 02:02:28.964599 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:02:28.905165    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:02:28.964599 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:28.937052    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg"]
Oct 31 02:02:28.964599 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:28.937194    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:02:28.964599 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:28.954166    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 31 02:02:28.964599 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:28.954220    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg" podUID="cd4d7677-6d53-42e9-8416-024a78dc3ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 31 02:02:28.964599 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:28.963019    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-c088fd\""
Oct 31 02:02:29.033117 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:29.032556    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc"]
Oct 31 02:02:29.099160 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:29.071307    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc"
Oct 31 02:02:29.194473 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:29.163097    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:29.240843 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:29.240744    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-c088fd-dockercfg-7jbl8\""
Oct 31 02:02:29.351063 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:29.348804    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft5nn\" (UniqueName: \"kubernetes.io/projected/c24d5105-6144-4a56-8aad-39be92445ea5-kube-api-access-ft5nn\") pod \"payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg\" (UID: \"c24d5105-6144-4a56-8aad-39be92445ea5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg"
Oct 31 02:02:29.351414 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:29.351386    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a1974b0-604c-4805-b965-0b0cfcf89b30-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc\" (UID: \"6a1974b0-604c-4805-b965-0b0cfcf89b30\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc"
Oct 31 02:02:29.351841 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:29.351810    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbjh4\" (UniqueName: \"kubernetes.io/projected/6a1974b0-604c-4805-b965-0b0cfcf89b30-kube-api-access-rbjh4\") pod \"payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc\" (UID: \"6a1974b0-604c-4805-b965-0b0cfcf89b30\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc"
Oct 31 02:02:29.352161 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:29.352138    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24d5105-6144-4a56-8aad-39be92445ea5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg\" (UID: \"c24d5105-6144-4a56-8aad-39be92445ea5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg"
Oct 31 02:02:29.456691 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:29.456657    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc"]
Oct 31 02:02:29.460733 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:29.460525    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:29.461016 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:29.460098    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft5nn\" (UniqueName: \"kubernetes.io/projected/c24d5105-6144-4a56-8aad-39be92445ea5-kube-api-access-ft5nn\") pod \"payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg\" (UID: \"c24d5105-6144-4a56-8aad-39be92445ea5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg"
Oct 31 02:02:29.466382 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:29.466094    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a1974b0-604c-4805-b965-0b0cfcf89b30-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc\" (UID: \"6a1974b0-604c-4805-b965-0b0cfcf89b30\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc"
Oct 31 02:02:29.476895 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:29.476863    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbjh4\" (UniqueName: \"kubernetes.io/projected/6a1974b0-604c-4805-b965-0b0cfcf89b30-kube-api-access-rbjh4\") pod \"payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc\" (UID: \"6a1974b0-604c-4805-b965-0b0cfcf89b30\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc"
Oct 31 02:02:29.487236 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:29.487204    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24d5105-6144-4a56-8aad-39be92445ea5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg\" (UID: \"c24d5105-6144-4a56-8aad-39be92445ea5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg"
Oct 31 02:02:29.488554 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:29.477332    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a1974b0-604c-4805-b965-0b0cfcf89b30-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc\" (UID: \"6a1974b0-604c-4805-b965-0b0cfcf89b30\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc"
Oct 31 02:02:29.490607 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:29.490579    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24d5105-6144-4a56-8aad-39be92445ea5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg\" (UID: \"c24d5105-6144-4a56-8aad-39be92445ea5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg"
Oct 31 02:02:29.531465 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:29.526628    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft5nn\" (UniqueName: \"kubernetes.io/projected/c24d5105-6144-4a56-8aad-39be92445ea5-kube-api-access-ft5nn\") pod \"payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg\" (UID: \"c24d5105-6144-4a56-8aad-39be92445ea5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg"
Oct 31 02:02:29.564483 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:29.562929    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbjh4\" (UniqueName: \"kubernetes.io/projected/6a1974b0-604c-4805-b965-0b0cfcf89b30-kube-api-access-rbjh4\") pod \"payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc\" (UID: \"6a1974b0-604c-4805-b965-0b0cfcf89b30\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc"
Oct 31 02:02:29.572845 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:29.572811    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:02:29.588437 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:29.588403    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc"
Oct 31 02:02:29.797535 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:29.796515    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg"
Oct 31 02:02:30.000474 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:29.988649    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 31 02:02:30.000474 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:29.988743    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg" podUID="cd4d7677-6d53-42e9-8416-024a78dc3ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 31 02:02:30.419648 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:30.419605    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc"]
Oct 31 02:02:30.692385 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:30.692221    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc" event={"ID":"6a1974b0-604c-4805-b965-0b0cfcf89b30","Type":"ContainerStarted","Data":"fbd013d074381a862662770f6ae7a48212f2205c6cac87980b18fc971d4a2083"}
Oct 31 02:02:30.792975 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:02:30.792929    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24d5105_6144_4a56_8aad_39be92445ea5.slice/crio-14a34086605cbc504c108934564de781a7b40e2f4f932f5ae28168ab6fb2ee88 WatchSource:0}: Error finding container 14a34086605cbc504c108934564de781a7b40e2f4f932f5ae28168ab6fb2ee88: Status 404 returned error can't find the container with id 14a34086605cbc504c108934564de781a7b40e2f4f932f5ae28168ab6fb2ee88
Oct 31 02:02:30.793354 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:30.793326    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg"]
Oct 31 02:02:30.948396 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:30.947885    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 31 02:02:30.948396 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:30.947946    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg" podUID="cd4d7677-6d53-42e9-8416-024a78dc3ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 31 02:02:31.713468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:31.713183    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg" event={"ID":"c24d5105-6144-4a56-8aad-39be92445ea5","Type":"ContainerStarted","Data":"f294c53ed00eb8a48dac3d12ab0b8e3a277f398c670d457cf9e423a9fc7a9d6e"}
Oct 31 02:02:31.713468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:31.713296    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg" event={"ID":"c24d5105-6144-4a56-8aad-39be92445ea5","Type":"ContainerStarted","Data":"14a34086605cbc504c108934564de781a7b40e2f4f932f5ae28168ab6fb2ee88"}
Oct 31 02:02:31.714042 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:31.713662    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg"
Oct 31 02:02:31.715508 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:31.715477    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc" event={"ID":"6a1974b0-604c-4805-b965-0b0cfcf89b30","Type":"ContainerStarted","Data":"2e06fbdd7d5a1688ac2c939b0f5d8c81211fa1f6d10390af57aa348b3c95d1a3"}
Oct 31 02:02:31.715626 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:31.715594    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc"
Oct 31 02:02:31.742895 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:31.742839    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg" podStartSLOduration=3.742824153 podStartE2EDuration="3.742824153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:31.738313926 +0000 UTC m=+2507.742028216" watchObservedRunningTime="2026-10-31 02:02:31.742824153 +0000 UTC m=+2507.746538444"
Oct 31 02:02:31.767463 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:31.767379    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc" podStartSLOduration=3.767356903 podStartE2EDuration="3.767356903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:31.765580853 +0000 UTC m=+2507.769295147" watchObservedRunningTime="2026-10-31 02:02:31.767356903 +0000 UTC m=+2507.771071195"
Oct 31 02:02:31.953886 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:31.953307    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:02:31.954301 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:31.954268    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:02:31.955324 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:31.955260    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:02:33.373304 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.373260    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"]
Oct 31 02:02:33.379050 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.379024    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.388114 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.388084    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"]
Oct 31 02:02:33.458669 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.458626    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.458669 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.458667    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.458934 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.458881    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.458990 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.458936    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.459042 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.459021    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.459090 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.459080    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.459207 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.459184    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.459268 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.459242    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.459320 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.459301    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzljs\" (UniqueName: \"kubernetes.io/projected/d7cacf9b-ad38-4f4b-a68c-f331dc576304-kube-api-access-rzljs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.560921 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.560881    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzljs\" (UniqueName: \"kubernetes.io/projected/d7cacf9b-ad38-4f4b-a68c-f331dc576304-kube-api-access-rzljs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.561108 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.561066    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.561108 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.561089    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.561778 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.561218    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.561778 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.561260    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.561778 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.561311    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.561778 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.561370    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.561778 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.561435    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.561778 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.561495    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.561778 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.561665    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.561778 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.561715    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.562084 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.561888    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.562084 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.561962    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.562299 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.562273    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.563783 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.563763    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.563895 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.563873    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.574963 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.574933    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.575090 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.575021    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzljs\" (UniqueName: \"kubernetes.io/projected/d7cacf9b-ad38-4f4b-a68c-f331dc576304-kube-api-access-rzljs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:33.697596 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:33.697511    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:34.011517 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:34.001470    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"]
Oct 31 02:02:34.011517 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:02:34.003346    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7cacf9b_ad38_4f4b_a68c_f331dc576304.slice/crio-919eb51cb3af56b5bd4a7e0c175c9bbc6ae4ae525a5a763ae94c6f39211214c9 WatchSource:0}: Error finding container 919eb51cb3af56b5bd4a7e0c175c9bbc6ae4ae525a5a763ae94c6f39211214c9: Status 404 returned error can't find the container with id 919eb51cb3af56b5bd4a7e0c175c9bbc6ae4ae525a5a763ae94c6f39211214c9
Oct 31 02:02:34.016077 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:34.016030    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:34.016235 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:34.016126    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:34.016235 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:34.016167    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:34.782025 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:34.781981    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm" event={"ID":"d7cacf9b-ad38-4f4b-a68c-f331dc576304","Type":"ContainerStarted","Data":"290f7a6869447094d79a15dce4826d7edbcad39b3e0d358f4c7824132dde4380"}
Oct 31 02:02:34.782594 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:34.782053    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm" event={"ID":"d7cacf9b-ad38-4f4b-a68c-f331dc576304","Type":"ContainerStarted","Data":"919eb51cb3af56b5bd4a7e0c175c9bbc6ae4ae525a5a763ae94c6f39211214c9"}
Oct 31 02:02:34.818010 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:34.817940    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm" podStartSLOduration=1.817566759 podStartE2EDuration="1.817566759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:34.813509244 +0000 UTC m=+2510.817223536" watchObservedRunningTime="2026-10-31 02:02:34.817566759 +0000 UTC m=+2510.821281053"
Oct 31 02:02:35.698432 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:35.698399    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:35.723357 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:35.723317    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 31 02:02:35.723683 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:35.723647    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm" podUID="d7cacf9b-ad38-4f4b-a68c-f331dc576304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 31 02:02:36.698439 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:36.698401    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 31 02:02:36.698858 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:36.698482    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm" podUID="d7cacf9b-ad38-4f4b-a68c-f331dc576304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 31 02:02:37.698530 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:37.698485    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 31 02:02:37.698956 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:37.698553    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm" podUID="d7cacf9b-ad38-4f4b-a68c-f331dc576304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 31 02:02:38.263493 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.263455    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"]
Oct 31 02:02:38.268732 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.268712    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.276233 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.276203    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0b6abf-openshift-default-dockercfg-xkp27\""
Oct 31 02:02:38.285813 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.283817    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"]
Oct 31 02:02:38.375915 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.375868    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-workload-socket\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.376132 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.376030    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0fa4248-e818-450c-bf1b-74f859787788-istiod-ca-cert\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.376132 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.376114    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrhl5\" (UniqueName: \"kubernetes.io/projected/e0fa4248-e818-450c-bf1b-74f859787788-kube-api-access-vrhl5\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.376315 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.376289    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0fa4248-e818-450c-bf1b-74f859787788-istio-podinfo\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.376563 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.376540    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-istio-envoy\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.376656 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.376583    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-credential-socket\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.376761 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.376743    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-workload-certs\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.376939 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.376920    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-istio-data\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.377017 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.377001    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0fa4248-e818-450c-bf1b-74f859787788-istio-token\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.479318 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.479279    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-workload-certs\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.479571 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.479396    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-istio-data\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.479571 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.479424    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0fa4248-e818-450c-bf1b-74f859787788-istio-token\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.479571 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.479516    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-workload-socket\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.479732 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.479588    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0fa4248-e818-450c-bf1b-74f859787788-istiod-ca-cert\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.479732 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.479653    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrhl5\" (UniqueName: \"kubernetes.io/projected/e0fa4248-e818-450c-bf1b-74f859787788-kube-api-access-vrhl5\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.479867 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.479754    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-workload-certs\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.479867 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.479831    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-istio-data\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.480004 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.479849    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0fa4248-e818-450c-bf1b-74f859787788-istio-podinfo\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.480004 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.479948    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-workload-socket\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.480114 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.480092    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-istio-envoy\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.480171 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.480132    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-credential-socket\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.480291 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.480268    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0fa4248-e818-450c-bf1b-74f859787788-istiod-ca-cert\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.480563 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.480544    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-credential-socket\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.482436 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.482415    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-istio-envoy\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.482602 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.482577    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0fa4248-e818-450c-bf1b-74f859787788-istio-podinfo\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.493366 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.493334    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0fa4248-e818-450c-bf1b-74f859787788-istio-token\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.493582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.493558    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrhl5\" (UniqueName: \"kubernetes.io/projected/e0fa4248-e818-450c-bf1b-74f859787788-kube-api-access-vrhl5\") pod \"e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") " pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.593428 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.593340    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:38.698614 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.698522    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 31 02:02:38.734516 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.698671    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm" podUID="d7cacf9b-ad38-4f4b-a68c-f331dc576304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 31 02:02:38.939061 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.939023    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"]
Oct 31 02:02:38.939842 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:02:38.939809    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0fa4248_e818_450c_bf1b_74f859787788.slice/crio-91a0be35cb2d3600d2cb53a8455d60dc1bc6a100463791dcce756e317c1ce838 WatchSource:0}: Error finding container 91a0be35cb2d3600d2cb53a8455d60dc1bc6a100463791dcce756e317c1ce838: Status 404 returned error can't find the container with id 91a0be35cb2d3600d2cb53a8455d60dc1bc6a100463791dcce756e317c1ce838
Oct 31 02:02:38.950654 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.946751    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:38.950654 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.946830    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:38.950654 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:38.946865    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:02:39.698177 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:39.698143    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 31 02:02:39.698367 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:39.698198    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm" podUID="d7cacf9b-ad38-4f4b-a68c-f331dc576304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 31 02:02:39.876608 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:39.876567    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr" event={"ID":"e0fa4248-e818-450c-bf1b-74f859787788","Type":"ContainerStarted","Data":"551e174065028f94449a6fa7da1b56f484f5e2157051a24e6a28b8e075e25d3b"}
Oct 31 02:02:39.876993 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:39.876625    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr" event={"ID":"e0fa4248-e818-450c-bf1b-74f859787788","Type":"ContainerStarted","Data":"91a0be35cb2d3600d2cb53a8455d60dc1bc6a100463791dcce756e317c1ce838"}
Oct 31 02:02:40.593875 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:40.593834    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:40.596348 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:40.596308    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 31 02:02:40.596513 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:40.596364    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr" podUID="e0fa4248-e818-450c-bf1b-74f859787788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 31 02:02:40.698794 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:40.698741    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 31 02:02:40.699018 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:40.698806    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm" podUID="d7cacf9b-ad38-4f4b-a68c-f331dc576304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 31 02:02:41.594245 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:41.594206    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 31 02:02:41.594718 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:41.594262    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr" podUID="e0fa4248-e818-450c-bf1b-74f859787788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 31 02:02:41.715471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:41.710658    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:41.719524 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:41.719165    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:41.731194 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:41.731163    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:02:41.756876 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:41.756809    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr" podStartSLOduration=3.756790393 podStartE2EDuration="3.756790393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:39.923692017 +0000 UTC m=+2515.927406309" watchObservedRunningTime="2026-10-31 02:02:41.756790393 +0000 UTC m=+2517.760504686"
Oct 31 02:02:42.594108 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:42.594068    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 31 02:02:42.594388 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:42.594133    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr" podUID="e0fa4248-e818-450c-bf1b-74f859787788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 31 02:02:42.728530 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:42.728491    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg"
Oct 31 02:02:42.728859 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:42.728838    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc"
Oct 31 02:02:43.594736 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:43.594699    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 31 02:02:43.595233 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:43.594754    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr" podUID="e0fa4248-e818-450c-bf1b-74f859787788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 31 02:02:44.594697 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:44.594667    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 31 02:02:44.594883 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:44.594718    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr" podUID="e0fa4248-e818-450c-bf1b-74f859787788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 31 02:02:45.593995 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:45.593953    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 31 02:02:45.594200 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:45.594011    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr" podUID="e0fa4248-e818-450c-bf1b-74f859787788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 31 02:02:46.608461 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:46.608414    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:46.611350 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:46.611320    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:46.612937 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:46.612916    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:02:46.915766 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:02:46.915669    5650 scope.go:122] "RemoveContainer" containerID="a80c4a3939ec818e1e72eb4c72d410e896f77ece8b7ff18d08bc8b243ab6dcaa"
Oct 31 02:02:58.561795 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:02:58.561694    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:03:08.683791 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:08.683745    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7"]
Oct 31 02:03:13.684396 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:13.684321    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7" podUID="bdb97f84-e14d-4bfc-a5ff-7b2041607ef8" containerName="payload-processing" containerID="cri-o://9d837ed1691b08b493fefe7a3f98002006d77fabcc2dd1694d0dae2dfc0e7d41" gracePeriod=25
Oct 31 02:03:14.031733 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:14.031702    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7"
Oct 31 02:03:14.135603 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:14.135563    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-provider-credentials\") pod \"bdb97f84-e14d-4bfc-a5ff-7b2041607ef8\" (UID: \"bdb97f84-e14d-4bfc-a5ff-7b2041607ef8\") "
Oct 31 02:03:14.135970 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:14.135943    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-routing-overlay-volume\") pod \"bdb97f84-e14d-4bfc-a5ff-7b2041607ef8\" (UID: \"bdb97f84-e14d-4bfc-a5ff-7b2041607ef8\") "
Oct 31 02:03:14.136283 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:14.136261    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-kube-api-access-ppbhc\" (UniqueName: \"kubernetes.io/projected/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-kube-api-access-ppbhc\") pod \"bdb97f84-e14d-4bfc-a5ff-7b2041607ef8\" (UID: \"bdb97f84-e14d-4bfc-a5ff-7b2041607ef8\") "
Oct 31 02:03:14.136615 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:14.136592    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-plugins-config-volume\") pod \"bdb97f84-e14d-4bfc-a5ff-7b2041607ef8\" (UID: \"bdb97f84-e14d-4bfc-a5ff-7b2041607ef8\") "
Oct 31 02:03:14.136908 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:14.136869    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-routing-overlay-volume" pod "bdb97f84-e14d-4bfc-a5ff-7b2041607ef8" (UID: "bdb97f84-e14d-4bfc-a5ff-7b2041607ef8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:03:14.138681 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:14.138656    5650 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-routing-overlay-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:03:14.139213 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:14.139190    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-plugins-config-volume" pod "bdb97f84-e14d-4bfc-a5ff-7b2041607ef8" (UID: "bdb97f84-e14d-4bfc-a5ff-7b2041607ef8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:03:14.157104 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:14.157045    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-provider-credentials" pod "bdb97f84-e14d-4bfc-a5ff-7b2041607ef8" (UID: "bdb97f84-e14d-4bfc-a5ff-7b2041607ef8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:03:14.159671 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:14.159633    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-kube-api-access-ppbhc" pod "bdb97f84-e14d-4bfc-a5ff-7b2041607ef8" (UID: "bdb97f84-e14d-4bfc-a5ff-7b2041607ef8"). InnerVolumeSpecName "kube-api-access-ppbhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:03:14.241656 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:14.241563    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:03:14.241656 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:14.241590    5650 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-provider-credentials\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:03:14.241656 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:14.241600    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppbhc\" (UniqueName: \"kubernetes.io/projected/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8-kube-api-access-ppbhc\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:03:14.506295 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:14.506201    5650 generic.go:356] "Generic (PLEG): container finished" podID="bdb97f84-e14d-4bfc-a5ff-7b2041607ef8" containerID="9d837ed1691b08b493fefe7a3f98002006d77fabcc2dd1694d0dae2dfc0e7d41" exitCode=0
Oct 31 02:03:14.506295 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:14.506263    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7"
Oct 31 02:03:14.506579 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:14.506289    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7" event={"ID":"bdb97f84-e14d-4bfc-a5ff-7b2041607ef8","Type":"ContainerDied","Data":"9d837ed1691b08b493fefe7a3f98002006d77fabcc2dd1694d0dae2dfc0e7d41"}
Oct 31 02:03:14.506579 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:14.506358    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7" event={"ID":"bdb97f84-e14d-4bfc-a5ff-7b2041607ef8","Type":"ContainerDied","Data":"019f7d43a691885c92db2ca1b9f7307252090e07bb117177363f3b31e1dbe535"}
Oct 31 02:03:14.506579 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:14.506390    5650 scope.go:122] "RemoveContainer" containerID="9d837ed1691b08b493fefe7a3f98002006d77fabcc2dd1694d0dae2dfc0e7d41"
Oct 31 02:03:14.519409 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:14.519391    5650 scope.go:122] "RemoveContainer" containerID="9d837ed1691b08b493fefe7a3f98002006d77fabcc2dd1694d0dae2dfc0e7d41"
Oct 31 02:03:14.519780 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:03:14.519758    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d837ed1691b08b493fefe7a3f98002006d77fabcc2dd1694d0dae2dfc0e7d41\": container with ID starting with 9d837ed1691b08b493fefe7a3f98002006d77fabcc2dd1694d0dae2dfc0e7d41 not found: ID does not exist" containerID="9d837ed1691b08b493fefe7a3f98002006d77fabcc2dd1694d0dae2dfc0e7d41"
Oct 31 02:03:14.519873 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:14.519788    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d837ed1691b08b493fefe7a3f98002006d77fabcc2dd1694d0dae2dfc0e7d41"} err="failed to get container status \"9d837ed1691b08b493fefe7a3f98002006d77fabcc2dd1694d0dae2dfc0e7d41\": rpc error: code = NotFound desc = could not find container \"9d837ed1691b08b493fefe7a3f98002006d77fabcc2dd1694d0dae2dfc0e7d41\": container with ID starting with 9d837ed1691b08b493fefe7a3f98002006d77fabcc2dd1694d0dae2dfc0e7d41 not found: ID does not exist"
Oct 31 02:03:14.563349 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:14.563317    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7"]
Oct 31 02:03:14.568169 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:14.568147    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-6b7f5mfj7"]
Oct 31 02:03:16.131414 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:16.131378    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdb97f84-e14d-4bfc-a5ff-7b2041607ef8" path="/var/lib/kubelet/pods/bdb97f84-e14d-4bfc-a5ff-7b2041607ef8/volumes"
Oct 31 02:03:28.557307 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:03:28.557260    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:03:36.858069 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:36.858027    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp"]
Oct 31 02:03:36.859191 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:36.859171    5650 cpu_manager.go:414] "containerMap: removing container" podUID="bdb97f84-e14d-4bfc-a5ff-7b2041607ef8" containerName="payload-processing"
Oct 31 02:03:36.859275 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:36.859192    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdb97f84-e14d-4bfc-a5ff-7b2041607ef8" containerName="payload-processing"
Oct 31 02:03:36.859389 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:36.859377    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdb97f84-e14d-4bfc-a5ff-7b2041607ef8" containerName="payload-processing"
Oct 31 02:03:36.997470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:36.963689    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp"]
Oct 31 02:03:36.997470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:36.963845    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp"
Oct 31 02:03:36.997470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:36.971809    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"payload-processing-external-model-plugins-e2e-extmod-a-28e4f3\""
Oct 31 02:03:37.120540 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:37.088233    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"payload-processing-external-model-e2e--1b1bd4a9-dockercfg-sxwc8\""
Oct 31 02:03:37.120540 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:37.088662    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"e2e-cmp-legacy-30b795-provider-cred\""
Oct 31 02:03:37.120540 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:37.104936    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"openshift-service-ca.crt\""
Oct 31 02:03:37.120540 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:37.106419    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"routing-overlay\""
Oct 31 02:03:37.120540 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:37.110931    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"kube-root-ca.crt\""
Oct 31 02:03:37.402712 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:37.402578    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfqhv\" (UniqueName: \"kubernetes.io/projected/7ee0352b-b1bd-4b10-854a-ab702cb8575b-kube-api-access-bfqhv\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp\" (UID: \"7ee0352b-b1bd-4b10-854a-ab702cb8575b\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp"
Oct 31 02:03:37.403187 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:37.403153    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ee0352b-b1bd-4b10-854a-ab702cb8575b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp\" (UID: \"7ee0352b-b1bd-4b10-854a-ab702cb8575b\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp"
Oct 31 02:03:37.403508 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:37.403425    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ee0352b-b1bd-4b10-854a-ab702cb8575b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp\" (UID: \"7ee0352b-b1bd-4b10-854a-ab702cb8575b\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp"
Oct 31 02:03:37.403668 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:37.403646    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee0352b-b1bd-4b10-854a-ab702cb8575b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp\" (UID: \"7ee0352b-b1bd-4b10-854a-ab702cb8575b\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp"
Oct 31 02:03:37.505173 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:37.505120    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ee0352b-b1bd-4b10-854a-ab702cb8575b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp\" (UID: \"7ee0352b-b1bd-4b10-854a-ab702cb8575b\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp"
Oct 31 02:03:37.505575 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:37.505345    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee0352b-b1bd-4b10-854a-ab702cb8575b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp\" (UID: \"7ee0352b-b1bd-4b10-854a-ab702cb8575b\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp"
Oct 31 02:03:37.505674 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:37.505566    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfqhv\" (UniqueName: \"kubernetes.io/projected/7ee0352b-b1bd-4b10-854a-ab702cb8575b-kube-api-access-bfqhv\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp\" (UID: \"7ee0352b-b1bd-4b10-854a-ab702cb8575b\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp"
Oct 31 02:03:37.505785 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:37.505762    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ee0352b-b1bd-4b10-854a-ab702cb8575b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp\" (UID: \"7ee0352b-b1bd-4b10-854a-ab702cb8575b\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp"
Oct 31 02:03:37.506325 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:37.506304    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee0352b-b1bd-4b10-854a-ab702cb8575b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp\" (UID: \"7ee0352b-b1bd-4b10-854a-ab702cb8575b\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp"
Oct 31 02:03:37.506413 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:37.506383    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ee0352b-b1bd-4b10-854a-ab702cb8575b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp\" (UID: \"7ee0352b-b1bd-4b10-854a-ab702cb8575b\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp"
Oct 31 02:03:37.514574 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:37.514542    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ee0352b-b1bd-4b10-854a-ab702cb8575b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp\" (UID: \"7ee0352b-b1bd-4b10-854a-ab702cb8575b\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp"
Oct 31 02:03:37.520015 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:37.519983    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfqhv\" (UniqueName: \"kubernetes.io/projected/7ee0352b-b1bd-4b10-854a-ab702cb8575b-kube-api-access-bfqhv\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp\" (UID: \"7ee0352b-b1bd-4b10-854a-ab702cb8575b\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp"
Oct 31 02:03:37.772929 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:37.772893    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp"
Oct 31 02:03:38.176745 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:03:38.176705    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee0352b_b1bd_4b10_854a_ab702cb8575b.slice/crio-8562b033dcb33319b0252131331b33407e3b5d525211bb44ed79a63894de0555 WatchSource:0}: Error finding container 8562b033dcb33319b0252131331b33407e3b5d525211bb44ed79a63894de0555: Status 404 returned error can't find the container with id 8562b033dcb33319b0252131331b33407e3b5d525211bb44ed79a63894de0555
Oct 31 02:03:38.181784 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:38.181738    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp"]
Oct 31 02:03:38.977546 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:38.974675    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp" event={"ID":"7ee0352b-b1bd-4b10-854a-ab702cb8575b","Type":"ContainerStarted","Data":"bce58502d4104cc27917ddd412761daa62f816f7ef75f6b597f1a9037cb2e15a"}
Oct 31 02:03:38.977546 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:38.974784    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp" event={"ID":"7ee0352b-b1bd-4b10-854a-ab702cb8575b","Type":"ContainerStarted","Data":"8562b033dcb33319b0252131331b33407e3b5d525211bb44ed79a63894de0555"}
Oct 31 02:03:38.977546 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:38.974817    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp"
Oct 31 02:03:39.011895 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:39.011827    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp" podStartSLOduration=3.011806032 podStartE2EDuration="3.011806032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:03:39.001261904 +0000 UTC m=+2575.004976198" watchObservedRunningTime="2026-10-31 02:03:39.011806032 +0000 UTC m=+2575.015520336"
Oct 31 02:03:39.942605 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:39.942567    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 02:03:39.943106 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:39.942633    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq" podUID="86d5dc42-ef1e-4ce5-9f29-386030d4524b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:03:49.089631 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:49.089584    5650 generic.go:356] "Generic (PLEG): container finished" podID="86d5dc42-ef1e-4ce5-9f29-386030d4524b" containerID="f3bfe1798fd1f9c7b6f9cc2d942897848f4e6ff1772c910fada3fced06f53a6e" exitCode=0
Oct 31 02:03:49.090924 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:49.089645    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq" event={"ID":"86d5dc42-ef1e-4ce5-9f29-386030d4524b","Type":"ContainerDied","Data":"f3bfe1798fd1f9c7b6f9cc2d942897848f4e6ff1772c910fada3fced06f53a6e"}
Oct 31 02:03:49.090924 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:49.090298    5650 scope.go:122] "RemoveContainer" containerID="f3bfe1798fd1f9c7b6f9cc2d942897848f4e6ff1772c910fada3fced06f53a6e"
Oct 31 02:03:49.092790 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:49.092752    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:03:49.092895 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:49.092846    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:03:49.092936 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:49.092892    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:03:49.990504 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:49.990469    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp"
Oct 31 02:03:50.197481 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:50.193500    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq" event={"ID":"86d5dc42-ef1e-4ce5-9f29-386030d4524b","Type":"ContainerStarted","Data":"6a7e86081199573084afa4897354324fe9671948e39d167df3500d57e861efce"}
Oct 31 02:03:50.975470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:50.975145    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:03:50.975470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:50.975284    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 31 02:03:50.975470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:50.975345    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq" podUID="86d5dc42-ef1e-4ce5-9f29-386030d4524b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 31 02:03:51.632795 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:51.626180    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Oct 31 02:03:51.944528 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:51.943550    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 02:03:51.978471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:51.973984    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 31 02:03:51.978471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:51.974078    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq" podUID="86d5dc42-ef1e-4ce5-9f29-386030d4524b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 31 02:03:51.978776 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:51.978569    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:03:52.015472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:52.011660    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:03:52.104593 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:52.104530    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:03:52.147635 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:52.147578    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:03:52.187814 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:52.187769    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 02:03:52.329470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:52.325745    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:03:52.970716 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:52.970677    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 31 02:03:52.971162 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:52.970748    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq" podUID="86d5dc42-ef1e-4ce5-9f29-386030d4524b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 31 02:03:53.889284 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:53.889253    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:03:53.971090 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:53.971056    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 31 02:03:53.971572 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:53.971109    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq" podUID="86d5dc42-ef1e-4ce5-9f29-386030d4524b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 31 02:03:54.980471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:54.978332    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 31 02:03:54.980471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:54.978481    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq" podUID="86d5dc42-ef1e-4ce5-9f29-386030d4524b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 31 02:03:55.976551 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:55.976516    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 31 02:03:55.977170 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:55.977129    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq" podUID="86d5dc42-ef1e-4ce5-9f29-386030d4524b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 31 02:03:56.970771 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:56.970731    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 31 02:03:56.971294 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:56.970801    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq" podUID="86d5dc42-ef1e-4ce5-9f29-386030d4524b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 31 02:03:57.977203 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:57.977170    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:03:57.978756 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:03:57.978732    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:03:58.567794 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:03:58.567757    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:04:28.557611 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:04:28.557521    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:04:32.367651 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:32.364501    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01be29b0-qll8s"]
Oct 31 02:04:32.407081 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:32.407053    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01be29b0-qll8s"
Oct 31 02:04:32.417769 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:32.417726    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:04:32.421645 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:32.421620    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:04:32.441899 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:32.441865    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01be29b0-qll8s"]
Oct 31 02:04:32.727473 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:32.712219    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cef28068-9242-4bdc-95dd-ea7620723963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01be29b0-qll8s\" (UID: \"cef28068-9242-4bdc-95dd-ea7620723963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01be29b0-qll8s"
Oct 31 02:04:32.742372 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:32.742311    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cef28068-9242-4bdc-95dd-ea7620723963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01be29b0-qll8s\" (UID: \"cef28068-9242-4bdc-95dd-ea7620723963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01be29b0-qll8s"
Oct 31 02:04:32.763475 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:32.763408    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k2dc\" (UniqueName: \"kubernetes.io/projected/cef28068-9242-4bdc-95dd-ea7620723963-kube-api-access-7k2dc\") pod \"maas-api-revoke-sub-01be29b0-qll8s\" (UID: \"cef28068-9242-4bdc-95dd-ea7620723963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01be29b0-qll8s"
Oct 31 02:04:32.867470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:32.865152    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k2dc\" (UniqueName: \"kubernetes.io/projected/cef28068-9242-4bdc-95dd-ea7620723963-kube-api-access-7k2dc\") pod \"maas-api-revoke-sub-01be29b0-qll8s\" (UID: \"cef28068-9242-4bdc-95dd-ea7620723963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01be29b0-qll8s"
Oct 31 02:04:32.867470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:32.865328    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cef28068-9242-4bdc-95dd-ea7620723963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01be29b0-qll8s\" (UID: \"cef28068-9242-4bdc-95dd-ea7620723963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01be29b0-qll8s"
Oct 31 02:04:32.867470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:32.865573    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cef28068-9242-4bdc-95dd-ea7620723963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01be29b0-qll8s\" (UID: \"cef28068-9242-4bdc-95dd-ea7620723963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01be29b0-qll8s"
Oct 31 02:04:32.867470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:32.867051    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cef28068-9242-4bdc-95dd-ea7620723963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01be29b0-qll8s\" (UID: \"cef28068-9242-4bdc-95dd-ea7620723963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01be29b0-qll8s"
Oct 31 02:04:32.870964 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:32.870931    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cef28068-9242-4bdc-95dd-ea7620723963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01be29b0-qll8s\" (UID: \"cef28068-9242-4bdc-95dd-ea7620723963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01be29b0-qll8s"
Oct 31 02:04:32.923052 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:32.923010    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k2dc\" (UniqueName: \"kubernetes.io/projected/cef28068-9242-4bdc-95dd-ea7620723963-kube-api-access-7k2dc\") pod \"maas-api-revoke-sub-01be29b0-qll8s\" (UID: \"cef28068-9242-4bdc-95dd-ea7620723963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01be29b0-qll8s"
Oct 31 02:04:33.027361 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:33.027273    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01be29b0-qll8s"
Oct 31 02:04:33.572667 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:33.570519    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01be29b0-qll8s"]
Oct 31 02:04:34.134160 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:34.134125    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01be29b0-qll8s" event={"ID":"cef28068-9242-4bdc-95dd-ea7620723963","Type":"ContainerStarted","Data":"c3c516c135d9251e0fc5bbcd08f287a171fd8826e927eb02fcb893b4372d40b4"}
Oct 31 02:04:35.145971 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:35.145934    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01be29b0-qll8s" event={"ID":"cef28068-9242-4bdc-95dd-ea7620723963","Type":"ContainerStarted","Data":"c9ae767bd1ed030cf3ceb458ed2102c9520b3f71bcd4b47083f7b96411646424"}
Oct 31 02:04:35.170908 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:35.170846    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01be29b0-qll8s" podStartSLOduration=3.170830064 podStartE2EDuration="3.170830064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:04:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:04:35.16631931 +0000 UTC m=+2631.170033602" watchObservedRunningTime="2026-10-31 02:04:35.170830064 +0000 UTC m=+2631.174544570"
Oct 31 02:04:36.123092 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:36.106824    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f45dfc5-8pnkk"]
Oct 31 02:04:36.123092 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:36.121848    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f45dfc5-8pnkk"
Oct 31 02:04:36.135952 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:36.135920    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f45dfc5-8pnkk"]
Oct 31 02:04:36.164546 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:36.164507    5650 generic.go:356] "Generic (PLEG): container finished" podID="cef28068-9242-4bdc-95dd-ea7620723963" containerID="c9ae767bd1ed030cf3ceb458ed2102c9520b3f71bcd4b47083f7b96411646424" exitCode=0
Oct 31 02:04:36.164974 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:36.164551    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01be29b0-qll8s" event={"ID":"cef28068-9242-4bdc-95dd-ea7620723963","Type":"ContainerDied","Data":"c9ae767bd1ed030cf3ceb458ed2102c9520b3f71bcd4b47083f7b96411646424"}
Oct 31 02:04:36.275857 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:36.275812    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f47ea99-7261-436b-a2d1-7ab1d54da8a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f45dfc5-8pnkk\" (UID: \"3f47ea99-7261-436b-a2d1-7ab1d54da8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f45dfc5-8pnkk"
Oct 31 02:04:36.276645 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:36.276609    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f47ea99-7261-436b-a2d1-7ab1d54da8a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f45dfc5-8pnkk\" (UID: \"3f47ea99-7261-436b-a2d1-7ab1d54da8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f45dfc5-8pnkk"
Oct 31 02:04:36.276963 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:36.276944    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qqnj\" (UniqueName: \"kubernetes.io/projected/3f47ea99-7261-436b-a2d1-7ab1d54da8a1-kube-api-access-9qqnj\") pod \"maas-api-revoke-sub-2f45dfc5-8pnkk\" (UID: \"3f47ea99-7261-436b-a2d1-7ab1d54da8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f45dfc5-8pnkk"
Oct 31 02:04:36.379112 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:36.379011    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qqnj\" (UniqueName: \"kubernetes.io/projected/3f47ea99-7261-436b-a2d1-7ab1d54da8a1-kube-api-access-9qqnj\") pod \"maas-api-revoke-sub-2f45dfc5-8pnkk\" (UID: \"3f47ea99-7261-436b-a2d1-7ab1d54da8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f45dfc5-8pnkk"
Oct 31 02:04:36.380069 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:36.379163    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f47ea99-7261-436b-a2d1-7ab1d54da8a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f45dfc5-8pnkk\" (UID: \"3f47ea99-7261-436b-a2d1-7ab1d54da8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f45dfc5-8pnkk"
Oct 31 02:04:36.380069 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:36.379502    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f47ea99-7261-436b-a2d1-7ab1d54da8a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f45dfc5-8pnkk\" (UID: \"3f47ea99-7261-436b-a2d1-7ab1d54da8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f45dfc5-8pnkk"
Oct 31 02:04:36.383114 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:36.383088    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f47ea99-7261-436b-a2d1-7ab1d54da8a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f45dfc5-8pnkk\" (UID: \"3f47ea99-7261-436b-a2d1-7ab1d54da8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f45dfc5-8pnkk"
Oct 31 02:04:36.389350 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:36.389318    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f47ea99-7261-436b-a2d1-7ab1d54da8a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f45dfc5-8pnkk\" (UID: \"3f47ea99-7261-436b-a2d1-7ab1d54da8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f45dfc5-8pnkk"
Oct 31 02:04:36.409921 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:36.409884    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qqnj\" (UniqueName: \"kubernetes.io/projected/3f47ea99-7261-436b-a2d1-7ab1d54da8a1-kube-api-access-9qqnj\") pod \"maas-api-revoke-sub-2f45dfc5-8pnkk\" (UID: \"3f47ea99-7261-436b-a2d1-7ab1d54da8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f45dfc5-8pnkk"
Oct 31 02:04:36.452685 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:36.452630    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f45dfc5-8pnkk"
Oct 31 02:04:36.859847 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:36.859816    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f45dfc5-8pnkk"]
Oct 31 02:04:36.862130 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:04:36.862096    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f47ea99_7261_436b_a2d1_7ab1d54da8a1.slice/crio-91665f5d7c9a611647cf8a9859b8d374a7655eded46f76ea808d0c34dd656737 WatchSource:0}: Error finding container 91665f5d7c9a611647cf8a9859b8d374a7655eded46f76ea808d0c34dd656737: Status 404 returned error can't find the container with id 91665f5d7c9a611647cf8a9859b8d374a7655eded46f76ea808d0c34dd656737
Oct 31 02:04:37.182549 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:37.182416    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f45dfc5-8pnkk" event={"ID":"3f47ea99-7261-436b-a2d1-7ab1d54da8a1","Type":"ContainerStarted","Data":"91665f5d7c9a611647cf8a9859b8d374a7655eded46f76ea808d0c34dd656737"}
Oct 31 02:04:37.383384 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:37.383355    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01be29b0-qll8s"
Oct 31 02:04:37.426303 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:37.426264    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cef28068-9242-4bdc-95dd-ea7620723963-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cef28068-9242-4bdc-95dd-ea7620723963-maas-api-xks-ca\") pod \"cef28068-9242-4bdc-95dd-ea7620723963\" (UID: \"cef28068-9242-4bdc-95dd-ea7620723963\") "
Oct 31 02:04:37.426545 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:37.426359    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cef28068-9242-4bdc-95dd-ea7620723963-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cef28068-9242-4bdc-95dd-ea7620723963-maas-api-openshift-ca\") pod \"cef28068-9242-4bdc-95dd-ea7620723963\" (UID: \"cef28068-9242-4bdc-95dd-ea7620723963\") "
Oct 31 02:04:37.426635 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:37.426610    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cef28068-9242-4bdc-95dd-ea7620723963-kube-api-access-7k2dc\" (UniqueName: \"kubernetes.io/projected/cef28068-9242-4bdc-95dd-ea7620723963-kube-api-access-7k2dc\") pod \"cef28068-9242-4bdc-95dd-ea7620723963\" (UID: \"cef28068-9242-4bdc-95dd-ea7620723963\") "
Oct 31 02:04:37.428156 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:37.428124    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cef28068-9242-4bdc-95dd-ea7620723963-maas-api-openshift-ca" pod "cef28068-9242-4bdc-95dd-ea7620723963" (UID: "cef28068-9242-4bdc-95dd-ea7620723963"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:04:37.429547 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:37.429528    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cef28068-9242-4bdc-95dd-ea7620723963-maas-api-xks-ca" pod "cef28068-9242-4bdc-95dd-ea7620723963" (UID: "cef28068-9242-4bdc-95dd-ea7620723963"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:04:37.437904 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:37.437827    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef28068-9242-4bdc-95dd-ea7620723963-kube-api-access-7k2dc" pod "cef28068-9242-4bdc-95dd-ea7620723963" (UID: "cef28068-9242-4bdc-95dd-ea7620723963"). InnerVolumeSpecName "kube-api-access-7k2dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:04:37.529265 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:37.529223    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cef28068-9242-4bdc-95dd-ea7620723963-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:37.529265 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:37.529258    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cef28068-9242-4bdc-95dd-ea7620723963-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:37.529265 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:37.529268    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k2dc\" (UniqueName: \"kubernetes.io/projected/cef28068-9242-4bdc-95dd-ea7620723963-kube-api-access-7k2dc\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:38.195461 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:38.195422    5650 generic.go:356] "Generic (PLEG): container finished" podID="3f47ea99-7261-436b-a2d1-7ab1d54da8a1" containerID="c66b6d9d57902a1de17f375c7e721db5e791dec06a8c50f5aa7bd04eee73f099" exitCode=0
Oct 31 02:04:38.195866 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:38.195480    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f45dfc5-8pnkk" event={"ID":"3f47ea99-7261-436b-a2d1-7ab1d54da8a1","Type":"ContainerDied","Data":"c66b6d9d57902a1de17f375c7e721db5e791dec06a8c50f5aa7bd04eee73f099"}
Oct 31 02:04:38.197152 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:38.197131    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01be29b0-qll8s"
Oct 31 02:04:38.197289 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:38.197165    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01be29b0-qll8s" event={"ID":"cef28068-9242-4bdc-95dd-ea7620723963","Type":"ContainerDied","Data":"c3c516c135d9251e0fc5bbcd08f287a171fd8826e927eb02fcb893b4372d40b4"}
Oct 31 02:04:38.197289 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:38.197218    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3c516c135d9251e0fc5bbcd08f287a171fd8826e927eb02fcb893b4372d40b4"
Oct 31 02:04:39.432388 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:39.432352    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f45dfc5-8pnkk"
Oct 31 02:04:39.601581 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:39.601500    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f47ea99-7261-436b-a2d1-7ab1d54da8a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f47ea99-7261-436b-a2d1-7ab1d54da8a1-maas-api-openshift-ca\") pod \"3f47ea99-7261-436b-a2d1-7ab1d54da8a1\" (UID: \"3f47ea99-7261-436b-a2d1-7ab1d54da8a1\") "
Oct 31 02:04:39.601581 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:39.601557    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f47ea99-7261-436b-a2d1-7ab1d54da8a1-kube-api-access-9qqnj\" (UniqueName: \"kubernetes.io/projected/3f47ea99-7261-436b-a2d1-7ab1d54da8a1-kube-api-access-9qqnj\") pod \"3f47ea99-7261-436b-a2d1-7ab1d54da8a1\" (UID: \"3f47ea99-7261-436b-a2d1-7ab1d54da8a1\") "
Oct 31 02:04:39.601786 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:39.601678    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f47ea99-7261-436b-a2d1-7ab1d54da8a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f47ea99-7261-436b-a2d1-7ab1d54da8a1-maas-api-xks-ca\") pod \"3f47ea99-7261-436b-a2d1-7ab1d54da8a1\" (UID: \"3f47ea99-7261-436b-a2d1-7ab1d54da8a1\") "
Oct 31 02:04:39.601950 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:39.601924    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f47ea99-7261-436b-a2d1-7ab1d54da8a1-maas-api-openshift-ca" pod "3f47ea99-7261-436b-a2d1-7ab1d54da8a1" (UID: "3f47ea99-7261-436b-a2d1-7ab1d54da8a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:04:39.602731 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:39.602706    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f47ea99-7261-436b-a2d1-7ab1d54da8a1-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:39.603796 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:39.603767    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f47ea99-7261-436b-a2d1-7ab1d54da8a1-maas-api-xks-ca" pod "3f47ea99-7261-436b-a2d1-7ab1d54da8a1" (UID: "3f47ea99-7261-436b-a2d1-7ab1d54da8a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:04:39.603895 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:39.603826    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f47ea99-7261-436b-a2d1-7ab1d54da8a1-kube-api-access-9qqnj" pod "3f47ea99-7261-436b-a2d1-7ab1d54da8a1" (UID: "3f47ea99-7261-436b-a2d1-7ab1d54da8a1"). InnerVolumeSpecName "kube-api-access-9qqnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:04:39.704277 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:39.704239    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qqnj\" (UniqueName: \"kubernetes.io/projected/3f47ea99-7261-436b-a2d1-7ab1d54da8a1-kube-api-access-9qqnj\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:39.704277 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:39.704274    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f47ea99-7261-436b-a2d1-7ab1d54da8a1-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:40.240460 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:40.240418    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f45dfc5-8pnkk"
Oct 31 02:04:40.240460 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:40.240441    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f45dfc5-8pnkk" event={"ID":"3f47ea99-7261-436b-a2d1-7ab1d54da8a1","Type":"ContainerDied","Data":"91665f5d7c9a611647cf8a9859b8d374a7655eded46f76ea808d0c34dd656737"}
Oct 31 02:04:40.240798 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:40.240499    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91665f5d7c9a611647cf8a9859b8d374a7655eded46f76ea808d0c34dd656737"
Oct 31 02:04:43.613665 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:43.613629    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da94df5d-gjgh6"]
Oct 31 02:04:43.614629 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:43.614611    5650 cpu_manager.go:414] "containerMap: removing container" podUID="3f47ea99-7261-436b-a2d1-7ab1d54da8a1" containerName="revoke-keys"
Oct 31 02:04:43.614691 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:43.614629    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f47ea99-7261-436b-a2d1-7ab1d54da8a1" containerName="revoke-keys"
Oct 31 02:04:43.614730 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:43.614692    5650 cpu_manager.go:414] "containerMap: removing container" podUID="cef28068-9242-4bdc-95dd-ea7620723963" containerName="revoke-keys"
Oct 31 02:04:43.614730 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:43.614697    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cef28068-9242-4bdc-95dd-ea7620723963" containerName="revoke-keys"
Oct 31 02:04:43.614847 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:43.614838    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="cef28068-9242-4bdc-95dd-ea7620723963" containerName="revoke-keys"
Oct 31 02:04:43.614887 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:43.614852    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f47ea99-7261-436b-a2d1-7ab1d54da8a1" containerName="revoke-keys"
Oct 31 02:04:43.618871 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:43.618809    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da94df5d-gjgh6"
Oct 31 02:04:43.623955 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:43.623930    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:04:43.625622 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:43.625596    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:04:43.625850 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:43.625827    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da94df5d-gjgh6"]
Oct 31 02:04:43.730961 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:43.730922    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18cc0eff-9057-45ef-8ee1-461d29daf8f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da94df5d-gjgh6\" (UID: \"18cc0eff-9057-45ef-8ee1-461d29daf8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da94df5d-gjgh6"
Oct 31 02:04:43.731136 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:43.731077    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98rgm\" (UniqueName: \"kubernetes.io/projected/18cc0eff-9057-45ef-8ee1-461d29daf8f9-kube-api-access-98rgm\") pod \"maas-api-revoke-sub-da94df5d-gjgh6\" (UID: \"18cc0eff-9057-45ef-8ee1-461d29daf8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da94df5d-gjgh6"
Oct 31 02:04:43.731300 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:43.731280    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18cc0eff-9057-45ef-8ee1-461d29daf8f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da94df5d-gjgh6\" (UID: \"18cc0eff-9057-45ef-8ee1-461d29daf8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da94df5d-gjgh6"
Oct 31 02:04:43.856473 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:43.851821    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18cc0eff-9057-45ef-8ee1-461d29daf8f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da94df5d-gjgh6\" (UID: \"18cc0eff-9057-45ef-8ee1-461d29daf8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da94df5d-gjgh6"
Oct 31 02:04:43.856473 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:43.851963    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98rgm\" (UniqueName: \"kubernetes.io/projected/18cc0eff-9057-45ef-8ee1-461d29daf8f9-kube-api-access-98rgm\") pod \"maas-api-revoke-sub-da94df5d-gjgh6\" (UID: \"18cc0eff-9057-45ef-8ee1-461d29daf8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da94df5d-gjgh6"
Oct 31 02:04:43.856473 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:43.852213    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18cc0eff-9057-45ef-8ee1-461d29daf8f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da94df5d-gjgh6\" (UID: \"18cc0eff-9057-45ef-8ee1-461d29daf8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da94df5d-gjgh6"
Oct 31 02:04:43.856796 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:43.856579    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18cc0eff-9057-45ef-8ee1-461d29daf8f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da94df5d-gjgh6\" (UID: \"18cc0eff-9057-45ef-8ee1-461d29daf8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da94df5d-gjgh6"
Oct 31 02:04:43.882523 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:43.880974    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18cc0eff-9057-45ef-8ee1-461d29daf8f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da94df5d-gjgh6\" (UID: \"18cc0eff-9057-45ef-8ee1-461d29daf8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da94df5d-gjgh6"
Oct 31 02:04:43.901869 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:43.900187    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98rgm\" (UniqueName: \"kubernetes.io/projected/18cc0eff-9057-45ef-8ee1-461d29daf8f9-kube-api-access-98rgm\") pod \"maas-api-revoke-sub-da94df5d-gjgh6\" (UID: \"18cc0eff-9057-45ef-8ee1-461d29daf8f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da94df5d-gjgh6"
Oct 31 02:04:43.945592 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:43.945557    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da94df5d-gjgh6"
Oct 31 02:04:44.249715 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:44.249681    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da94df5d-gjgh6"]
Oct 31 02:04:44.250501 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:04:44.250472    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18cc0eff_9057_45ef_8ee1_461d29daf8f9.slice/crio-cec65a60bba0335a9eb79013ac67689574e16cc2c157f9f136044d946c8bd35d WatchSource:0}: Error finding container cec65a60bba0335a9eb79013ac67689574e16cc2c157f9f136044d946c8bd35d: Status 404 returned error can't find the container with id cec65a60bba0335a9eb79013ac67689574e16cc2c157f9f136044d946c8bd35d
Oct 31 02:04:44.282832 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:44.282799    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da94df5d-gjgh6" event={"ID":"18cc0eff-9057-45ef-8ee1-461d29daf8f9","Type":"ContainerStarted","Data":"cec65a60bba0335a9eb79013ac67689574e16cc2c157f9f136044d946c8bd35d"}
Oct 31 02:04:45.294269 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:45.294175    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da94df5d-gjgh6" event={"ID":"18cc0eff-9057-45ef-8ee1-461d29daf8f9","Type":"ContainerStarted","Data":"00641a5ffe18415594361184c220ce59d37aa421613aba5e13292f6ef2309414"}
Oct 31 02:04:45.319175 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:45.319125    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da94df5d-gjgh6" podStartSLOduration=2.31910994 podStartE2EDuration="2.31910994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:04:45.314394663 +0000 UTC m=+2641.318108954" watchObservedRunningTime="2026-10-31 02:04:45.31910994 +0000 UTC m=+2641.322824243"
Oct 31 02:04:46.306260 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:46.306224    5650 generic.go:356] "Generic (PLEG): container finished" podID="18cc0eff-9057-45ef-8ee1-461d29daf8f9" containerID="00641a5ffe18415594361184c220ce59d37aa421613aba5e13292f6ef2309414" exitCode=0
Oct 31 02:04:46.306716 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:46.306311    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da94df5d-gjgh6" event={"ID":"18cc0eff-9057-45ef-8ee1-461d29daf8f9","Type":"ContainerDied","Data":"00641a5ffe18415594361184c220ce59d37aa421613aba5e13292f6ef2309414"}
Oct 31 02:04:47.872649 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:47.869773    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da94df5d-gjgh6"
Oct 31 02:04:48.024722 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:48.024685    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18cc0eff-9057-45ef-8ee1-461d29daf8f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18cc0eff-9057-45ef-8ee1-461d29daf8f9-maas-api-openshift-ca\") pod \"18cc0eff-9057-45ef-8ee1-461d29daf8f9\" (UID: \"18cc0eff-9057-45ef-8ee1-461d29daf8f9\") "
Oct 31 02:04:48.024939 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:48.024734    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18cc0eff-9057-45ef-8ee1-461d29daf8f9-kube-api-access-98rgm\" (UniqueName: \"kubernetes.io/projected/18cc0eff-9057-45ef-8ee1-461d29daf8f9-kube-api-access-98rgm\") pod \"18cc0eff-9057-45ef-8ee1-461d29daf8f9\" (UID: \"18cc0eff-9057-45ef-8ee1-461d29daf8f9\") "
Oct 31 02:04:48.024939 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:48.024763    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18cc0eff-9057-45ef-8ee1-461d29daf8f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18cc0eff-9057-45ef-8ee1-461d29daf8f9-maas-api-xks-ca\") pod \"18cc0eff-9057-45ef-8ee1-461d29daf8f9\" (UID: \"18cc0eff-9057-45ef-8ee1-461d29daf8f9\") "
Oct 31 02:04:48.025118 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:48.025090    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18cc0eff-9057-45ef-8ee1-461d29daf8f9-maas-api-openshift-ca" pod "18cc0eff-9057-45ef-8ee1-461d29daf8f9" (UID: "18cc0eff-9057-45ef-8ee1-461d29daf8f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:04:48.025673 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:48.025648    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18cc0eff-9057-45ef-8ee1-461d29daf8f9-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:48.026980 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:48.026951    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18cc0eff-9057-45ef-8ee1-461d29daf8f9-kube-api-access-98rgm" pod "18cc0eff-9057-45ef-8ee1-461d29daf8f9" (UID: "18cc0eff-9057-45ef-8ee1-461d29daf8f9"). InnerVolumeSpecName "kube-api-access-98rgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:04:48.027070 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:48.027008    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18cc0eff-9057-45ef-8ee1-461d29daf8f9-maas-api-xks-ca" pod "18cc0eff-9057-45ef-8ee1-461d29daf8f9" (UID: "18cc0eff-9057-45ef-8ee1-461d29daf8f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:04:48.127120 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:48.127086    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98rgm\" (UniqueName: \"kubernetes.io/projected/18cc0eff-9057-45ef-8ee1-461d29daf8f9-kube-api-access-98rgm\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:48.127120 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:48.127119    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18cc0eff-9057-45ef-8ee1-461d29daf8f9-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:04:48.332849 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:48.332771    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da94df5d-gjgh6"
Oct 31 02:04:48.332849 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:48.332809    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da94df5d-gjgh6" event={"ID":"18cc0eff-9057-45ef-8ee1-461d29daf8f9","Type":"ContainerDied","Data":"cec65a60bba0335a9eb79013ac67689574e16cc2c157f9f136044d946c8bd35d"}
Oct 31 02:04:48.333040 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:04:48.332856    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cec65a60bba0335a9eb79013ac67689574e16cc2c157f9f136044d946c8bd35d"
Oct 31 02:04:58.554975 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:04:58.554937    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:05:14.325034 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:14.324983    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp"]
Oct 31 02:05:19.336293 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:19.336243    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp" podUID="7ee0352b-b1bd-4b10-854a-ab702cb8575b" containerName="payload-processing" containerID="cri-o://bce58502d4104cc27917ddd412761daa62f816f7ef75f6b597f1a9037cb2e15a" gracePeriod=25
Oct 31 02:05:20.195709 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.194995    5650 generic.go:356] "Generic (PLEG): container finished" podID="7ee0352b-b1bd-4b10-854a-ab702cb8575b" containerID="bce58502d4104cc27917ddd412761daa62f816f7ef75f6b597f1a9037cb2e15a" exitCode=0
Oct 31 02:05:20.197867 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.197827    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp" event={"ID":"7ee0352b-b1bd-4b10-854a-ab702cb8575b","Type":"ContainerDied","Data":"bce58502d4104cc27917ddd412761daa62f816f7ef75f6b597f1a9037cb2e15a"}
Oct 31 02:05:20.198040 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.197903    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp" event={"ID":"7ee0352b-b1bd-4b10-854a-ab702cb8575b","Type":"ContainerDied","Data":"8562b033dcb33319b0252131331b33407e3b5d525211bb44ed79a63894de0555"}
Oct 31 02:05:20.198040 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.197929    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8562b033dcb33319b0252131331b33407e3b5d525211bb44ed79a63894de0555"
Oct 31 02:05:20.215464 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.211885    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp"
Oct 31 02:05:20.257383 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.255103    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ee0352b-b1bd-4b10-854a-ab702cb8575b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee0352b-b1bd-4b10-854a-ab702cb8575b-plugins-config-volume\") pod \"7ee0352b-b1bd-4b10-854a-ab702cb8575b\" (UID: \"7ee0352b-b1bd-4b10-854a-ab702cb8575b\") "
Oct 31 02:05:20.257383 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.255328    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee0352b-b1bd-4b10-854a-ab702cb8575b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ee0352b-b1bd-4b10-854a-ab702cb8575b-provider-credentials\") pod \"7ee0352b-b1bd-4b10-854a-ab702cb8575b\" (UID: \"7ee0352b-b1bd-4b10-854a-ab702cb8575b\") "
Oct 31 02:05:20.257383 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.255488    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee0352b-b1bd-4b10-854a-ab702cb8575b-kube-api-access-bfqhv\" (UniqueName: \"kubernetes.io/projected/7ee0352b-b1bd-4b10-854a-ab702cb8575b-kube-api-access-bfqhv\") pod \"7ee0352b-b1bd-4b10-854a-ab702cb8575b\" (UID: \"7ee0352b-b1bd-4b10-854a-ab702cb8575b\") "
Oct 31 02:05:20.257383 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.255631    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ee0352b-b1bd-4b10-854a-ab702cb8575b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ee0352b-b1bd-4b10-854a-ab702cb8575b-routing-overlay-volume\") pod \"7ee0352b-b1bd-4b10-854a-ab702cb8575b\" (UID: \"7ee0352b-b1bd-4b10-854a-ab702cb8575b\") "
Oct 31 02:05:20.283520 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.266821    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee0352b-b1bd-4b10-854a-ab702cb8575b-routing-overlay-volume" pod "7ee0352b-b1bd-4b10-854a-ab702cb8575b" (UID: "7ee0352b-b1bd-4b10-854a-ab702cb8575b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:20.283520 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.268135    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee0352b-b1bd-4b10-854a-ab702cb8575b-plugins-config-volume" pod "7ee0352b-b1bd-4b10-854a-ab702cb8575b" (UID: "7ee0352b-b1bd-4b10-854a-ab702cb8575b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:20.335364 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.329523    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx"]
Oct 31 02:05:20.335364 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.330922    5650 cpu_manager.go:414] "containerMap: removing container" podUID="7ee0352b-b1bd-4b10-854a-ab702cb8575b" containerName="payload-processing"
Oct 31 02:05:20.335364 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.330945    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee0352b-b1bd-4b10-854a-ab702cb8575b" containerName="payload-processing"
Oct 31 02:05:20.335364 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.330964    5650 cpu_manager.go:414] "containerMap: removing container" podUID="18cc0eff-9057-45ef-8ee1-461d29daf8f9" containerName="revoke-keys"
Oct 31 02:05:20.335364 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.330974    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18cc0eff-9057-45ef-8ee1-461d29daf8f9" containerName="revoke-keys"
Oct 31 02:05:20.335364 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.331237    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee0352b-b1bd-4b10-854a-ab702cb8575b" containerName="payload-processing"
Oct 31 02:05:20.335364 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.331257    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="18cc0eff-9057-45ef-8ee1-461d29daf8f9" containerName="revoke-keys"
Oct 31 02:05:20.377340 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.369681    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee0352b-b1bd-4b10-854a-ab702cb8575b-provider-credentials" pod "7ee0352b-b1bd-4b10-854a-ab702cb8575b" (UID: "7ee0352b-b1bd-4b10-854a-ab702cb8575b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:20.377340 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.369794    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee0352b-b1bd-4b10-854a-ab702cb8575b-kube-api-access-bfqhv" pod "7ee0352b-b1bd-4b10-854a-ab702cb8575b" (UID: "7ee0352b-b1bd-4b10-854a-ab702cb8575b"). InnerVolumeSpecName "kube-api-access-bfqhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:20.382069 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.382042    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx"]
Oct 31 02:05:20.382412 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.382328    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx"
Oct 31 02:05:20.383217 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.383166    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee0352b-b1bd-4b10-854a-ab702cb8575b-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:20.383374 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.383358    5650 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ee0352b-b1bd-4b10-854a-ab702cb8575b-provider-credentials\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:20.383567 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.383515    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfqhv\" (UniqueName: \"kubernetes.io/projected/7ee0352b-b1bd-4b10-854a-ab702cb8575b-kube-api-access-bfqhv\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:20.383959 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.383939    5650 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ee0352b-b1bd-4b10-854a-ab702cb8575b-routing-overlay-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:20.393298 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.390843    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"payload-processing-external-model-e2e--1b1bd4a9-dockercfg-7r8fh\""
Oct 31 02:05:20.393298 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.391008    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"agc-cred-290407c1d817\""
Oct 31 02:05:20.393298 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.391072    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"e2e-cfl-cec4cc-provider-cred\""
Oct 31 02:05:20.527468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.521770    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx\" (UID: \"4bb7e02b-ec7b-4c37-8cee-aa73396e4500\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx"
Oct 31 02:05:20.527468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.521921    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx\" (UID: \"4bb7e02b-ec7b-4c37-8cee-aa73396e4500\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx"
Oct 31 02:05:20.527468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.522051    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx\" (UID: \"4bb7e02b-ec7b-4c37-8cee-aa73396e4500\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx"
Oct 31 02:05:20.527468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.522267    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwsxk\" (UniqueName: \"kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-kube-api-access-gwsxk\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx\" (UID: \"4bb7e02b-ec7b-4c37-8cee-aa73396e4500\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx"
Oct 31 02:05:20.700530 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.698230    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwsxk\" (UniqueName: \"kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-kube-api-access-gwsxk\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx\" (UID: \"4bb7e02b-ec7b-4c37-8cee-aa73396e4500\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx"
Oct 31 02:05:20.711971 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.710582    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwsxk\" (UniqueName: \"kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-kube-api-access-gwsxk\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx\" (UID: \"4bb7e02b-ec7b-4c37-8cee-aa73396e4500\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx"
Oct 31 02:05:20.711971 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.710811    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx\" (UID: \"4bb7e02b-ec7b-4c37-8cee-aa73396e4500\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx"
Oct 31 02:05:20.711971 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.710962    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx\" (UID: \"4bb7e02b-ec7b-4c37-8cee-aa73396e4500\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx"
Oct 31 02:05:20.711971 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.711138    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx\" (UID: \"4bb7e02b-ec7b-4c37-8cee-aa73396e4500\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx"
Oct 31 02:05:20.711971 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:05:20.711417    5650 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-28e4f3/agc-cred-290407c1d817: secret "agc-cred-290407c1d817" not found
Oct 31 02:05:20.711971 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:05:20.711459    5650 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx: secret "agc-cred-290407c1d817" not found
Oct 31 02:05:20.711971 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:05:20.711531    5650 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-provider-credentials podName:4bb7e02b-ec7b-4c37-8cee-aa73396e4500 nodeName:}" failed. No retries permitted until 2026-10-31 02:05:21.211503581 +0000 UTC m=+2677.215217862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx" (UID: "4bb7e02b-ec7b-4c37-8cee-aa73396e4500") : secret "agc-cred-290407c1d817" not found
Oct 31 02:05:20.714100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.714045    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx\" (UID: \"4bb7e02b-ec7b-4c37-8cee-aa73396e4500\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx"
Oct 31 02:05:20.714774 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.714751    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx\" (UID: \"4bb7e02b-ec7b-4c37-8cee-aa73396e4500\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx"
Oct 31 02:05:20.987934 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:20.987880    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp" podUID="7ee0352b-b1bd-4b10-854a-ab702cb8575b" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.225:50052\" within 1s: context deadline exceeded"
Oct 31 02:05:20.988165 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:05:20.987974    5650 logging.go:55] [core] [Channel #5192 SubChannel #5193]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.225:50052", ServerName: "10.134.0.225:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.225:50052: operation was canceled"
Oct 31 02:05:21.211777 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:21.211353    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp"
Oct 31 02:05:21.224056 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:21.223403    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx\" (UID: \"4bb7e02b-ec7b-4c37-8cee-aa73396e4500\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx"
Oct 31 02:05:21.225826 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:05:21.225800    5650 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-28e4f3/agc-cred-290407c1d817: secret "agc-cred-290407c1d817" not found
Oct 31 02:05:21.225826 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:05:21.225828    5650 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx: secret "agc-cred-290407c1d817" not found
Oct 31 02:05:21.226026 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:05:21.225884    5650 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-provider-credentials podName:4bb7e02b-ec7b-4c37-8cee-aa73396e4500 nodeName:}" failed. No retries permitted until 2026-10-31 02:05:22.225864651 +0000 UTC m=+2678.229578921 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx" (UID: "4bb7e02b-ec7b-4c37-8cee-aa73396e4500") : secret "agc-cred-290407c1d817" not found
Oct 31 02:05:21.260562 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:21.258313    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp"]
Oct 31 02:05:21.263789 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:21.263753    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5cd9xqfdp"]
Oct 31 02:05:22.189162 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:22.189109    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee0352b-b1bd-4b10-854a-ab702cb8575b" path="/var/lib/kubelet/pods/7ee0352b-b1bd-4b10-854a-ab702cb8575b/volumes"
Oct 31 02:05:22.357471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:22.356282    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx\" (UID: \"4bb7e02b-ec7b-4c37-8cee-aa73396e4500\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx"
Oct 31 02:05:22.365407 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:05:22.358288    5650 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-28e4f3/agc-cred-290407c1d817: secret "agc-cred-290407c1d817" not found
Oct 31 02:05:22.365407 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:05:22.358316    5650 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx: secret "agc-cred-290407c1d817" not found
Oct 31 02:05:22.365407 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:05:22.358610    5650 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-provider-credentials podName:4bb7e02b-ec7b-4c37-8cee-aa73396e4500 nodeName:}" failed. No retries permitted until 2026-10-31 02:05:24.358587119 +0000 UTC m=+2680.362301391 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx" (UID: "4bb7e02b-ec7b-4c37-8cee-aa73396e4500") : secret "agc-cred-290407c1d817" not found
Oct 31 02:05:24.408005 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:24.407968    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx\" (UID: \"4bb7e02b-ec7b-4c37-8cee-aa73396e4500\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx"
Oct 31 02:05:24.408427 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:05:24.408121    5650 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-28e4f3/agc-cred-290407c1d817: secret "agc-cred-290407c1d817" not found
Oct 31 02:05:24.408427 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:05:24.408148    5650 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx: secret "agc-cred-290407c1d817" not found
Oct 31 02:05:24.408427 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:05:24.408218    5650 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-provider-credentials podName:4bb7e02b-ec7b-4c37-8cee-aa73396e4500 nodeName:}" failed. No retries permitted until 2026-10-31 02:05:28.408201529 +0000 UTC m=+2684.411915814 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx" (UID: "4bb7e02b-ec7b-4c37-8cee-aa73396e4500") : secret "agc-cred-290407c1d817" not found
Oct 31 02:05:26.100682 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:26.100641    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"]
Oct 31 02:05:26.101164 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:26.100897    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj" podUID="cd6018a4-4e7e-410a-892f-6b3f800addd7" containerName="istio-proxy" containerID="cri-o://95e8a7b8df9be5a647234ed5aed5b7341d4b12f4cb18b647fdfd5e03349c45e5" gracePeriod=30
Oct 31 02:05:28.494033 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:28.493991    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx\" (UID: \"4bb7e02b-ec7b-4c37-8cee-aa73396e4500\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx"
Oct 31 02:05:28.494436 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:05:28.494145    5650 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-28e4f3/agc-cred-290407c1d817: secret "agc-cred-290407c1d817" not found
Oct 31 02:05:28.494436 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:05:28.494169    5650 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx: secret "agc-cred-290407c1d817" not found
Oct 31 02:05:28.494436 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:05:28.494235    5650 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-provider-credentials podName:4bb7e02b-ec7b-4c37-8cee-aa73396e4500 nodeName:}" failed. No retries permitted until 2026-10-31 02:05:36.494219572 +0000 UTC m=+2692.497933841 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx" (UID: "4bb7e02b-ec7b-4c37-8cee-aa73396e4500") : secret "agc-cred-290407c1d817" not found
Oct 31 02:05:28.556212 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:05:28.556171    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:05:31.203460 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:05:31.203407    5650 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd6018a4_4e7e_410a_892f_6b3f800addd7.slice/crio-95e8a7b8df9be5a647234ed5aed5b7341d4b12f4cb18b647fdfd5e03349c45e5.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 02:05:31.414639 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.414607    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 02:05:31.420150 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.420114    5650 generic.go:356] "Generic (PLEG): container finished" podID="cd6018a4-4e7e-410a-892f-6b3f800addd7" containerID="95e8a7b8df9be5a647234ed5aed5b7341d4b12f4cb18b647fdfd5e03349c45e5" exitCode=0
Oct 31 02:05:31.420292 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.420165    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"
Oct 31 02:05:31.420292 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.420190    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj" event={"ID":"cd6018a4-4e7e-410a-892f-6b3f800addd7","Type":"ContainerDied","Data":"95e8a7b8df9be5a647234ed5aed5b7341d4b12f4cb18b647fdfd5e03349c45e5"}
Oct 31 02:05:31.420292 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.420270    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj" event={"ID":"cd6018a4-4e7e-410a-892f-6b3f800addd7","Type":"ContainerDied","Data":"3a5cb11b4bb9d3888d545e219c2bba2e101d019dd1ba21f017792e0f881fcaeb"}
Oct 31 02:05:31.420292 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.420292    5650 scope.go:122] "RemoveContainer" containerID="95e8a7b8df9be5a647234ed5aed5b7341d4b12f4cb18b647fdfd5e03349c45e5"
Oct 31 02:05:31.439116 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.439094    5650 scope.go:122] "RemoveContainer" containerID="ef3c2fc9fedc601a7e8676e784a2668eca04022ab412a13713042dde38b7b5e7"
Oct 31 02:05:31.457639 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.457605    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg"]
Oct 31 02:05:31.459033 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.459001    5650 cpu_manager.go:414] "containerMap: removing container" podUID="cd6018a4-4e7e-410a-892f-6b3f800addd7" containerName="istio-proxy"
Oct 31 02:05:31.459033 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.459027    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd6018a4-4e7e-410a-892f-6b3f800addd7" containerName="istio-proxy"
Oct 31 02:05:31.459194 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.459110    5650 cpu_manager.go:414] "containerMap: removing container" podUID="cd6018a4-4e7e-410a-892f-6b3f800addd7" containerName="istio-proxy"
Oct 31 02:05:31.459194 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.459119    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd6018a4-4e7e-410a-892f-6b3f800addd7" containerName="istio-proxy"
Oct 31 02:05:31.459336 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.459323    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd6018a4-4e7e-410a-892f-6b3f800addd7" containerName="istio-proxy"
Oct 31 02:05:31.459376 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.459347    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd6018a4-4e7e-410a-892f-6b3f800addd7" containerName="istio-proxy"
Oct 31 02:05:31.461605 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.461580    5650 scope.go:122] "RemoveContainer" containerID="95e8a7b8df9be5a647234ed5aed5b7341d4b12f4cb18b647fdfd5e03349c45e5"
Oct 31 02:05:31.462146 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:05:31.462003    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e8a7b8df9be5a647234ed5aed5b7341d4b12f4cb18b647fdfd5e03349c45e5\": container with ID starting with 95e8a7b8df9be5a647234ed5aed5b7341d4b12f4cb18b647fdfd5e03349c45e5 not found: ID does not exist" containerID="95e8a7b8df9be5a647234ed5aed5b7341d4b12f4cb18b647fdfd5e03349c45e5"
Oct 31 02:05:31.462146 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.462046    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e8a7b8df9be5a647234ed5aed5b7341d4b12f4cb18b647fdfd5e03349c45e5"} err="failed to get container status \"95e8a7b8df9be5a647234ed5aed5b7341d4b12f4cb18b647fdfd5e03349c45e5\": rpc error: code = NotFound desc = could not find container \"95e8a7b8df9be5a647234ed5aed5b7341d4b12f4cb18b647fdfd5e03349c45e5\": container with ID starting with 95e8a7b8df9be5a647234ed5aed5b7341d4b12f4cb18b647fdfd5e03349c45e5 not found: ID does not exist"
Oct 31 02:05:31.462146 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.462066    5650 scope.go:122] "RemoveContainer" containerID="ef3c2fc9fedc601a7e8676e784a2668eca04022ab412a13713042dde38b7b5e7"
Oct 31 02:05:31.462419 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:05:31.462388    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef3c2fc9fedc601a7e8676e784a2668eca04022ab412a13713042dde38b7b5e7\": container with ID starting with ef3c2fc9fedc601a7e8676e784a2668eca04022ab412a13713042dde38b7b5e7 not found: ID does not exist" containerID="ef3c2fc9fedc601a7e8676e784a2668eca04022ab412a13713042dde38b7b5e7"
Oct 31 02:05:31.462542 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.462418    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef3c2fc9fedc601a7e8676e784a2668eca04022ab412a13713042dde38b7b5e7"} err="failed to get container status \"ef3c2fc9fedc601a7e8676e784a2668eca04022ab412a13713042dde38b7b5e7\": rpc error: code = NotFound desc = could not find container \"ef3c2fc9fedc601a7e8676e784a2668eca04022ab412a13713042dde38b7b5e7\": container with ID starting with ef3c2fc9fedc601a7e8676e784a2668eca04022ab412a13713042dde38b7b5e7 not found: ID does not exist"
Oct 31 02:05:31.465352 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.465332    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp"]
Oct 31 02:05:31.465624 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.465505    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg"
Oct 31 02:05:31.478097 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.478072    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ac7e0d\"/\"kube-root-ca.crt\""
Oct 31 02:05:31.479591 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.479228    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ac7e0d\"/\"agc-cred-717e7914d161\""
Oct 31 02:05:31.479739 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.479308    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ac7e0d\"/\"agc-cred-cb4b5707c14d\""
Oct 31 02:05:31.479880 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.479431    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ac7e0d\"/\"payload-processing-external-model-e2e--1ed73e85-dockercfg-cd74l\""
Oct 31 02:05:31.480026 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.479542    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ac7e0d\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ac7e0d\""
Oct 31 02:05:31.480122 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.479545    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ac7e0d\"/\"openshift-service-ca.crt\""
Oct 31 02:05:31.480122 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.479624    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ac7e0d\"/\"routing-overlay\""
Oct 31 02:05:31.484656 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.483738    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg"]
Oct 31 02:05:31.484656 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.483908    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp"
Oct 31 02:05:31.489466 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.489412    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp"]
Oct 31 02:05:31.569591 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.569471    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd6018a4-4e7e-410a-892f-6b3f800addd7-kube-api-access-lzzb4\" (UniqueName: \"kubernetes.io/projected/cd6018a4-4e7e-410a-892f-6b3f800addd7-kube-api-access-lzzb4\") pod \"cd6018a4-4e7e-410a-892f-6b3f800addd7\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") "
Oct 31 02:05:31.569591 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.569537    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-credential-socket\") pod \"cd6018a4-4e7e-410a-892f-6b3f800addd7\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") "
Oct 31 02:05:31.569858 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.569680    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-podinfo\") pod \"cd6018a4-4e7e-410a-892f-6b3f800addd7\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") "
Oct 31 02:05:31.569858 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.569765    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-workload-certs\") pod \"cd6018a4-4e7e-410a-892f-6b3f800addd7\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") "
Oct 31 02:05:31.569961 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.569864    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-data\") pod \"cd6018a4-4e7e-410a-892f-6b3f800addd7\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") "
Oct 31 02:05:31.570069 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.570039    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-envoy\") pod \"cd6018a4-4e7e-410a-892f-6b3f800addd7\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") "
Oct 31 02:05:31.570293 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.570272    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-token\" (UniqueName: \"kubernetes.io/projected/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-token\") pod \"cd6018a4-4e7e-410a-892f-6b3f800addd7\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") "
Oct 31 02:05:31.570411 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.570336    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-workload-socket\") pod \"cd6018a4-4e7e-410a-892f-6b3f800addd7\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") "
Oct 31 02:05:31.570411 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.570404    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd6018a4-4e7e-410a-892f-6b3f800addd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd6018a4-4e7e-410a-892f-6b3f800addd7-istiod-ca-cert\") pod \"cd6018a4-4e7e-410a-892f-6b3f800addd7\" (UID: \"cd6018a4-4e7e-410a-892f-6b3f800addd7\") "
Oct 31 02:05:31.570603 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.570422    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-credential-socket" pod "cd6018a4-4e7e-410a-892f-6b3f800addd7" (UID: "cd6018a4-4e7e-410a-892f-6b3f800addd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:31.570867 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.570839    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg\" (UID: \"bb268bae-9d2f-493a-81e5-fdc55ebd9b11\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg"
Oct 31 02:05:31.571045 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.571029    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59c4ccc1-c5bd-4666-a461-7802d5773296-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp\" (UID: \"59c4ccc1-c5bd-4666-a461-7802d5773296\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp"
Oct 31 02:05:31.571143 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.571131    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlxv9\" (UniqueName: \"kubernetes.io/projected/59c4ccc1-c5bd-4666-a461-7802d5773296-kube-api-access-tlxv9\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp\" (UID: \"59c4ccc1-c5bd-4666-a461-7802d5773296\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp"
Oct 31 02:05:31.571211 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.571200    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/59c4ccc1-c5bd-4666-a461-7802d5773296-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp\" (UID: \"59c4ccc1-c5bd-4666-a461-7802d5773296\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp"
Oct 31 02:05:31.571277 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.571247    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg\" (UID: \"bb268bae-9d2f-493a-81e5-fdc55ebd9b11\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg"
Oct 31 02:05:31.571277 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.571267    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg\" (UID: \"bb268bae-9d2f-493a-81e5-fdc55ebd9b11\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg"
Oct 31 02:05:31.571424 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.571385    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/59c4ccc1-c5bd-4666-a461-7802d5773296-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp\" (UID: \"59c4ccc1-c5bd-4666-a461-7802d5773296\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp"
Oct 31 02:05:31.571544 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.571467    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts5xp\" (UniqueName: \"kubernetes.io/projected/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-kube-api-access-ts5xp\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg\" (UID: \"bb268bae-9d2f-493a-81e5-fdc55ebd9b11\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg"
Oct 31 02:05:31.571631 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.571612    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:31.572493 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.572428    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-data" pod "cd6018a4-4e7e-410a-892f-6b3f800addd7" (UID: "cd6018a4-4e7e-410a-892f-6b3f800addd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:31.573284 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.572523    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-workload-socket" pod "cd6018a4-4e7e-410a-892f-6b3f800addd7" (UID: "cd6018a4-4e7e-410a-892f-6b3f800addd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:31.573284 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.572979    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-workload-certs" pod "cd6018a4-4e7e-410a-892f-6b3f800addd7" (UID: "cd6018a4-4e7e-410a-892f-6b3f800addd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:31.574851 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.574825    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-envoy" pod "cd6018a4-4e7e-410a-892f-6b3f800addd7" (UID: "cd6018a4-4e7e-410a-892f-6b3f800addd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:31.574851 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.574836    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-token" pod "cd6018a4-4e7e-410a-892f-6b3f800addd7" (UID: "cd6018a4-4e7e-410a-892f-6b3f800addd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:31.575597 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.575573    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd6018a4-4e7e-410a-892f-6b3f800addd7-istiod-ca-cert" pod "cd6018a4-4e7e-410a-892f-6b3f800addd7" (UID: "cd6018a4-4e7e-410a-892f-6b3f800addd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:31.575597 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.575586    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-podinfo" pod "cd6018a4-4e7e-410a-892f-6b3f800addd7" (UID: "cd6018a4-4e7e-410a-892f-6b3f800addd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:05:31.600329 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.600275    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd6018a4-4e7e-410a-892f-6b3f800addd7-kube-api-access-lzzb4" pod "cd6018a4-4e7e-410a-892f-6b3f800addd7" (UID: "cd6018a4-4e7e-410a-892f-6b3f800addd7"). InnerVolumeSpecName "kube-api-access-lzzb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:31.678594 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.677565    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlxv9\" (UniqueName: \"kubernetes.io/projected/59c4ccc1-c5bd-4666-a461-7802d5773296-kube-api-access-tlxv9\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp\" (UID: \"59c4ccc1-c5bd-4666-a461-7802d5773296\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp"
Oct 31 02:05:31.678594 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.677693    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/59c4ccc1-c5bd-4666-a461-7802d5773296-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp\" (UID: \"59c4ccc1-c5bd-4666-a461-7802d5773296\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp"
Oct 31 02:05:31.678594 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.677761    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg\" (UID: \"bb268bae-9d2f-493a-81e5-fdc55ebd9b11\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg"
Oct 31 02:05:31.678594 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.677785    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg\" (UID: \"bb268bae-9d2f-493a-81e5-fdc55ebd9b11\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg"
Oct 31 02:05:31.678594 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.677899    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/59c4ccc1-c5bd-4666-a461-7802d5773296-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp\" (UID: \"59c4ccc1-c5bd-4666-a461-7802d5773296\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp"
Oct 31 02:05:31.678594 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.677970    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts5xp\" (UniqueName: \"kubernetes.io/projected/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-kube-api-access-ts5xp\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg\" (UID: \"bb268bae-9d2f-493a-81e5-fdc55ebd9b11\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg"
Oct 31 02:05:31.678594 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.678084    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg\" (UID: \"bb268bae-9d2f-493a-81e5-fdc55ebd9b11\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg"
Oct 31 02:05:31.688021 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.679931    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg\" (UID: \"bb268bae-9d2f-493a-81e5-fdc55ebd9b11\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg"
Oct 31 02:05:31.688021 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.683088    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/59c4ccc1-c5bd-4666-a461-7802d5773296-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp\" (UID: \"59c4ccc1-c5bd-4666-a461-7802d5773296\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp"
Oct 31 02:05:31.688021 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.685390    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59c4ccc1-c5bd-4666-a461-7802d5773296-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp\" (UID: \"59c4ccc1-c5bd-4666-a461-7802d5773296\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp"
Oct 31 02:05:31.688021 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.687800    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd6018a4-4e7e-410a-892f-6b3f800addd7-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:31.688658 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.688400    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg\" (UID: \"bb268bae-9d2f-493a-81e5-fdc55ebd9b11\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg"
Oct 31 02:05:31.688658 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.688459    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzzb4\" (UniqueName: \"kubernetes.io/projected/cd6018a4-4e7e-410a-892f-6b3f800addd7-kube-api-access-lzzb4\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:31.688658 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.688480    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:31.688658 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.688496    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:31.688658 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.688512    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:31.688658 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.688527    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:31.688658 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.688541    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd6018a4-4e7e-410a-892f-6b3f800addd7-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:31.688658 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.688558    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6018a4-4e7e-410a-892f-6b3f800addd7-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:31.714733 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.689922    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59c4ccc1-c5bd-4666-a461-7802d5773296-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp\" (UID: \"59c4ccc1-c5bd-4666-a461-7802d5773296\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp"
Oct 31 02:05:31.714733 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.694391    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlxv9\" (UniqueName: \"kubernetes.io/projected/59c4ccc1-c5bd-4666-a461-7802d5773296-kube-api-access-tlxv9\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp\" (UID: \"59c4ccc1-c5bd-4666-a461-7802d5773296\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp"
Oct 31 02:05:31.714733 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.703846    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/59c4ccc1-c5bd-4666-a461-7802d5773296-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp\" (UID: \"59c4ccc1-c5bd-4666-a461-7802d5773296\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp"
Oct 31 02:05:31.714733 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.710147    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg\" (UID: \"bb268bae-9d2f-493a-81e5-fdc55ebd9b11\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg"
Oct 31 02:05:31.737492 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.732217    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts5xp\" (UniqueName: \"kubernetes.io/projected/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-kube-api-access-ts5xp\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg\" (UID: \"bb268bae-9d2f-493a-81e5-fdc55ebd9b11\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg"
Oct 31 02:05:31.775208 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.775164    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"]
Oct 31 02:05:31.781585 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.781549    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-w2ptj"]
Oct 31 02:05:31.794008 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.793975    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg"
Oct 31 02:05:31.813656 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:31.813621    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp"
Oct 31 02:05:32.088355 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:05:32.088324    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb268bae_9d2f_493a_81e5_fdc55ebd9b11.slice/crio-f162655181a961fb94b20bea279c7c6b3aa600f128c1c631a196870a93dbd457 WatchSource:0}: Error finding container f162655181a961fb94b20bea279c7c6b3aa600f128c1c631a196870a93dbd457: Status 404 returned error can't find the container with id f162655181a961fb94b20bea279c7c6b3aa600f128c1c631a196870a93dbd457
Oct 31 02:05:32.089552 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:32.088801    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg"]
Oct 31 02:05:32.113347 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:05:32.113303    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c4ccc1_c5bd_4666_a461_7802d5773296.slice/crio-eb807048c8eac6f1bdf1460abe7df7c6fe06787902cc9f41860def38651d88f4 WatchSource:0}: Error finding container eb807048c8eac6f1bdf1460abe7df7c6fe06787902cc9f41860def38651d88f4: Status 404 returned error can't find the container with id eb807048c8eac6f1bdf1460abe7df7c6fe06787902cc9f41860def38651d88f4
Oct 31 02:05:32.126773 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:32.126741    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd6018a4-4e7e-410a-892f-6b3f800addd7" path="/var/lib/kubelet/pods/cd6018a4-4e7e-410a-892f-6b3f800addd7/volumes"
Oct 31 02:05:32.128650 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:32.128628    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp"]
Oct 31 02:05:32.434543 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:32.434404    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp" event={"ID":"59c4ccc1-c5bd-4666-a461-7802d5773296","Type":"ContainerStarted","Data":"7d9d72459f28ab1d9fa58792e5aa17ae9635484e220c328eca57bcd00467ed25"}
Oct 31 02:05:32.434543 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:32.434489    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp"
Oct 31 02:05:32.434543 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:32.434519    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp" event={"ID":"59c4ccc1-c5bd-4666-a461-7802d5773296","Type":"ContainerStarted","Data":"eb807048c8eac6f1bdf1460abe7df7c6fe06787902cc9f41860def38651d88f4"}
Oct 31 02:05:32.436315 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:32.436286    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg" event={"ID":"bb268bae-9d2f-493a-81e5-fdc55ebd9b11","Type":"ContainerStarted","Data":"4d72f2c3278518da5912ab8bad71e06aaeb4e8eaf6970b42e5792ba503039f6a"}
Oct 31 02:05:32.436547 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:32.436346    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg" event={"ID":"bb268bae-9d2f-493a-81e5-fdc55ebd9b11","Type":"ContainerStarted","Data":"f162655181a961fb94b20bea279c7c6b3aa600f128c1c631a196870a93dbd457"}
Oct 31 02:05:32.436547 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:32.436380    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg"
Oct 31 02:05:32.471935 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:32.471856    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp" podStartSLOduration=9.471832938 podStartE2EDuration="9.471832938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:05:32.458283469 +0000 UTC m=+2688.461997761" watchObservedRunningTime="2026-10-31 02:05:32.471832938 +0000 UTC m=+2688.475547229"
Oct 31 02:05:32.495840 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:32.495782    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg" podStartSLOduration=8.495767466 podStartE2EDuration="8.495767466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:05:32.492625607 +0000 UTC m=+2688.496339898" watchObservedRunningTime="2026-10-31 02:05:32.495767466 +0000 UTC m=+2688.499481757"
Oct 31 02:05:36.589818 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:36.589766    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx\" (UID: \"4bb7e02b-ec7b-4c37-8cee-aa73396e4500\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx"
Oct 31 02:05:36.590260 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:05:36.589919    5650 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-28e4f3/agc-cred-290407c1d817: secret "agc-cred-290407c1d817" not found
Oct 31 02:05:36.590260 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:05:36.589945    5650 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx: secret "agc-cred-290407c1d817" not found
Oct 31 02:05:36.590260 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:05:36.590017    5650 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-provider-credentials podName:4bb7e02b-ec7b-4c37-8cee-aa73396e4500 nodeName:}" failed. No retries permitted until 2026-10-31 02:05:52.590000201 +0000 UTC m=+2708.593714487 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx" (UID: "4bb7e02b-ec7b-4c37-8cee-aa73396e4500") : secret "agc-cred-290407c1d817" not found
Oct 31 02:05:37.364565 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:37.364520    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx"]
Oct 31 02:05:37.365356 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:05:37.365319    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx" podUID="4bb7e02b-ec7b-4c37-8cee-aa73396e4500"
Oct 31 02:05:37.547423 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:37.547389    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx"
Oct 31 02:05:37.577947 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:37.574330    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx"
Oct 31 02:05:37.610992 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:37.610958    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-plugins-config-volume\") pod \"4bb7e02b-ec7b-4c37-8cee-aa73396e4500\" (UID: \"4bb7e02b-ec7b-4c37-8cee-aa73396e4500\") "
Oct 31 02:05:37.611604 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:37.611259    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-kube-api-access-gwsxk\" (UniqueName: \"kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-kube-api-access-gwsxk\") pod \"4bb7e02b-ec7b-4c37-8cee-aa73396e4500\" (UID: \"4bb7e02b-ec7b-4c37-8cee-aa73396e4500\") "
Oct 31 02:05:37.611604 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:37.611367    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-routing-overlay-volume\") pod \"4bb7e02b-ec7b-4c37-8cee-aa73396e4500\" (UID: \"4bb7e02b-ec7b-4c37-8cee-aa73396e4500\") "
Oct 31 02:05:37.611604 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:37.611422    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-plugins-config-volume" pod "4bb7e02b-ec7b-4c37-8cee-aa73396e4500" (UID: "4bb7e02b-ec7b-4c37-8cee-aa73396e4500"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:37.611811 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:37.611777    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-routing-overlay-volume" pod "4bb7e02b-ec7b-4c37-8cee-aa73396e4500" (UID: "4bb7e02b-ec7b-4c37-8cee-aa73396e4500"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:37.612388 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:37.612365    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:37.612535 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:37.612390    5650 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-routing-overlay-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:37.613722 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:37.613700    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-kube-api-access-gwsxk" pod "4bb7e02b-ec7b-4c37-8cee-aa73396e4500" (UID: "4bb7e02b-ec7b-4c37-8cee-aa73396e4500"). InnerVolumeSpecName "kube-api-access-gwsxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:37.724468 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:37.723885    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwsxk\" (UniqueName: \"kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-kube-api-access-gwsxk\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:38.559159 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.559120    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx"
Oct 31 02:05:38.614026 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.613975    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx"]
Oct 31 02:05:38.620514 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.620482    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj"]
Oct 31 02:05:38.626431 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.626400    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj"
Oct 31 02:05:38.626965 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.626941    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-555c5zxsx"]
Oct 31 02:05:38.634305 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.634277    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-df9d42\"/\"agc-cred-cfe3b0811863\""
Oct 31 02:05:38.639642 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.638115    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-df9d42\"/\"kube-root-ca.crt\""
Oct 31 02:05:38.639642 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.638234    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-df9d42\"/\"openshift-service-ca.crt\""
Oct 31 02:05:38.639642 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.638315    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-df9d42\"/\"payload-processing-external-model-plugins-e2e-extmod-b-df9d42\""
Oct 31 02:05:38.639642 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.638126    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-df9d42\"/\"routing-overlay\""
Oct 31 02:05:38.648202 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.648168    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj"]
Oct 31 02:05:38.743467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.743413    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-df9d42\"/\"payload-processing-external-model-e2e--da998f83-dockercfg-5f7c7\""
Oct 31 02:05:38.756645 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.756599    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6bcb\" (UniqueName: \"kubernetes.io/projected/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-kube-api-access-t6bcb\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj\" (UID: \"4caa846a-c5cf-467c-a0d7-8889e60c5bbf\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj"
Oct 31 02:05:38.756857 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.756690    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj\" (UID: \"4caa846a-c5cf-467c-a0d7-8889e60c5bbf\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj"
Oct 31 02:05:38.757096 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.757074    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj\" (UID: \"4caa846a-c5cf-467c-a0d7-8889e60c5bbf\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj"
Oct 31 02:05:38.757202 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.757157    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj\" (UID: \"4caa846a-c5cf-467c-a0d7-8889e60c5bbf\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj"
Oct 31 02:05:38.757409 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.757392    5650 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4bb7e02b-ec7b-4c37-8cee-aa73396e4500-provider-credentials\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:38.859596 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.859490    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj\" (UID: \"4caa846a-c5cf-467c-a0d7-8889e60c5bbf\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj"
Oct 31 02:05:38.859839 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.859593    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj\" (UID: \"4caa846a-c5cf-467c-a0d7-8889e60c5bbf\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj"
Oct 31 02:05:38.859839 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.859706    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6bcb\" (UniqueName: \"kubernetes.io/projected/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-kube-api-access-t6bcb\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj\" (UID: \"4caa846a-c5cf-467c-a0d7-8889e60c5bbf\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj"
Oct 31 02:05:38.859839 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.859742    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj\" (UID: \"4caa846a-c5cf-467c-a0d7-8889e60c5bbf\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj"
Oct 31 02:05:38.861090 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.861062    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj\" (UID: \"4caa846a-c5cf-467c-a0d7-8889e60c5bbf\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj"
Oct 31 02:05:38.861263 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.861237    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj\" (UID: \"4caa846a-c5cf-467c-a0d7-8889e60c5bbf\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj"
Oct 31 02:05:38.862399 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.862375    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj\" (UID: \"4caa846a-c5cf-467c-a0d7-8889e60c5bbf\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj"
Oct 31 02:05:38.873652 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.873615    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6bcb\" (UniqueName: \"kubernetes.io/projected/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-kube-api-access-t6bcb\") pod \"payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj\" (UID: \"4caa846a-c5cf-467c-a0d7-8889e60c5bbf\") " pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj"
Oct 31 02:05:38.952267 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:38.952227    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj"
Oct 31 02:05:39.252693 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:39.252639    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj"]
Oct 31 02:05:39.253292 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:05:39.253251    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4caa846a_c5cf_467c_a0d7_8889e60c5bbf.slice/crio-2f7a754b3e25d31a0abe71020cb20eb6e46d19660a5000248a7785c5c05c32a1 WatchSource:0}: Error finding container 2f7a754b3e25d31a0abe71020cb20eb6e46d19660a5000248a7785c5c05c32a1: Status 404 returned error can't find the container with id 2f7a754b3e25d31a0abe71020cb20eb6e46d19660a5000248a7785c5c05c32a1
Oct 31 02:05:39.570269 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:39.570171    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj" event={"ID":"4caa846a-c5cf-467c-a0d7-8889e60c5bbf","Type":"ContainerStarted","Data":"78ea270dd1625606cdca7385e73b1aed91c42ed20f4b2efbd6ed99f4776ab238"}
Oct 31 02:05:39.570269 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:39.570260    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj" event={"ID":"4caa846a-c5cf-467c-a0d7-8889e60c5bbf","Type":"ContainerStarted","Data":"2f7a754b3e25d31a0abe71020cb20eb6e46d19660a5000248a7785c5c05c32a1"}
Oct 31 02:05:39.570591 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:39.570390    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj"
Oct 31 02:05:39.595830 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:39.595779    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj" podStartSLOduration=2.59576481 podStartE2EDuration="2.59576481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:05:39.594580939 +0000 UTC m=+2695.598295231" watchObservedRunningTime="2026-10-31 02:05:39.59576481 +0000 UTC m=+2695.599479101"
Oct 31 02:05:40.123161 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:40.123125    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bb7e02b-ec7b-4c37-8cee-aa73396e4500" path="/var/lib/kubelet/pods/4bb7e02b-ec7b-4c37-8cee-aa73396e4500/volumes"
Oct 31 02:05:43.514473 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:43.495683    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg"
Oct 31 02:05:43.514473 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:43.503159    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp" podUID="59c4ccc1-c5bd-4666-a461-7802d5773296" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:05:43.797582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:43.794435    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp"]
Oct 31 02:05:43.826804 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:43.826758    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp" podUID="59c4ccc1-c5bd-4666-a461-7802d5773296" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:05:45.008225 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:45.008188    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:05:48.795815 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:48.795761    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp" podUID="59c4ccc1-c5bd-4666-a461-7802d5773296" containerName="payload-processing" containerID="cri-o://7d9d72459f28ab1d9fa58792e5aa17ae9635484e220c328eca57bcd00467ed25" gracePeriod=25
Oct 31 02:05:48.920747 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:48.920717    5650 generic.go:356] "Generic (PLEG): container finished" podID="59c4ccc1-c5bd-4666-a461-7802d5773296" containerID="7d9d72459f28ab1d9fa58792e5aa17ae9635484e220c328eca57bcd00467ed25" exitCode=0
Oct 31 02:05:48.920885 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:48.920783    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp" event={"ID":"59c4ccc1-c5bd-4666-a461-7802d5773296","Type":"ContainerDied","Data":"7d9d72459f28ab1d9fa58792e5aa17ae9635484e220c328eca57bcd00467ed25"}
Oct 31 02:05:49.120537 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:49.120508    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp"
Oct 31 02:05:49.226467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:49.224501    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59c4ccc1-c5bd-4666-a461-7802d5773296-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59c4ccc1-c5bd-4666-a461-7802d5773296-plugins-config-volume\") pod \"59c4ccc1-c5bd-4666-a461-7802d5773296\" (UID: \"59c4ccc1-c5bd-4666-a461-7802d5773296\") "
Oct 31 02:05:49.226467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:49.224615    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59c4ccc1-c5bd-4666-a461-7802d5773296-kube-api-access-tlxv9\" (UniqueName: \"kubernetes.io/projected/59c4ccc1-c5bd-4666-a461-7802d5773296-kube-api-access-tlxv9\") pod \"59c4ccc1-c5bd-4666-a461-7802d5773296\" (UID: \"59c4ccc1-c5bd-4666-a461-7802d5773296\") "
Oct 31 02:05:49.226467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:49.224750    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59c4ccc1-c5bd-4666-a461-7802d5773296-provider-credentials\" (UniqueName: \"kubernetes.io/projected/59c4ccc1-c5bd-4666-a461-7802d5773296-provider-credentials\") pod \"59c4ccc1-c5bd-4666-a461-7802d5773296\" (UID: \"59c4ccc1-c5bd-4666-a461-7802d5773296\") "
Oct 31 02:05:49.226467 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:49.224844    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59c4ccc1-c5bd-4666-a461-7802d5773296-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/59c4ccc1-c5bd-4666-a461-7802d5773296-routing-overlay-volume\") pod \"59c4ccc1-c5bd-4666-a461-7802d5773296\" (UID: \"59c4ccc1-c5bd-4666-a461-7802d5773296\") "
Oct 31 02:05:49.228331 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:49.227995    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c4ccc1-c5bd-4666-a461-7802d5773296-plugins-config-volume" pod "59c4ccc1-c5bd-4666-a461-7802d5773296" (UID: "59c4ccc1-c5bd-4666-a461-7802d5773296"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:49.237953 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:49.237913    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c4ccc1-c5bd-4666-a461-7802d5773296-routing-overlay-volume" pod "59c4ccc1-c5bd-4666-a461-7802d5773296" (UID: "59c4ccc1-c5bd-4666-a461-7802d5773296"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:49.240480 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:49.240430    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c4ccc1-c5bd-4666-a461-7802d5773296-kube-api-access-tlxv9" pod "59c4ccc1-c5bd-4666-a461-7802d5773296" (UID: "59c4ccc1-c5bd-4666-a461-7802d5773296"). InnerVolumeSpecName "kube-api-access-tlxv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:49.240626 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:49.240488    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c4ccc1-c5bd-4666-a461-7802d5773296-provider-credentials" pod "59c4ccc1-c5bd-4666-a461-7802d5773296" (UID: "59c4ccc1-c5bd-4666-a461-7802d5773296"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:49.330049 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:49.329965    5650 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/59c4ccc1-c5bd-4666-a461-7802d5773296-provider-credentials\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:49.330049 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:49.329999    5650 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/59c4ccc1-c5bd-4666-a461-7802d5773296-routing-overlay-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:49.330049 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:49.330014    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59c4ccc1-c5bd-4666-a461-7802d5773296-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:49.330049 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:49.330027    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlxv9\" (UniqueName: \"kubernetes.io/projected/59c4ccc1-c5bd-4666-a461-7802d5773296-kube-api-access-tlxv9\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:49.510847 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:49.510810    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj"]
Oct 31 02:05:49.513994 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:49.513957    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj"
Oct 31 02:05:50.239544 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:50.235505    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp"
Oct 31 02:05:50.273661 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:50.259335    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp" event={"ID":"59c4ccc1-c5bd-4666-a461-7802d5773296","Type":"ContainerDied","Data":"eb807048c8eac6f1bdf1460abe7df7c6fe06787902cc9f41860def38651d88f4"}
Oct 31 02:05:50.280386 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:50.280355    5650 scope.go:122] "RemoveContainer" containerID="7d9d72459f28ab1d9fa58792e5aa17ae9635484e220c328eca57bcd00467ed25"
Oct 31 02:05:50.363227 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:50.363189    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp"]
Oct 31 02:05:50.372136 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:50.372103    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-64f6rdzjp"]
Oct 31 02:05:52.123866 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:52.123830    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59c4ccc1-c5bd-4666-a461-7802d5773296" path="/var/lib/kubelet/pods/59c4ccc1-c5bd-4666-a461-7802d5773296/volumes"
Oct 31 02:05:54.511779 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:54.511735    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj" podUID="4caa846a-c5cf-467c-a0d7-8889e60c5bbf" containerName="payload-processing" containerID="cri-o://78ea270dd1625606cdca7385e73b1aed91c42ed20f4b2efbd6ed99f4776ab238" gracePeriod=25
Oct 31 02:05:54.841954 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:54.841922    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj"
Oct 31 02:05:54.878082 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:54.878040    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"]
Oct 31 02:05:54.879483 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:54.879437    5650 cpu_manager.go:414] "containerMap: removing container" podUID="4caa846a-c5cf-467c-a0d7-8889e60c5bbf" containerName="payload-processing"
Oct 31 02:05:54.879483 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:54.879482    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4caa846a-c5cf-467c-a0d7-8889e60c5bbf" containerName="payload-processing"
Oct 31 02:05:54.879672 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:54.879525    5650 cpu_manager.go:414] "containerMap: removing container" podUID="59c4ccc1-c5bd-4666-a461-7802d5773296" containerName="payload-processing"
Oct 31 02:05:54.879672 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:54.879535    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59c4ccc1-c5bd-4666-a461-7802d5773296" containerName="payload-processing"
Oct 31 02:05:54.879852 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:54.879834    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="4caa846a-c5cf-467c-a0d7-8889e60c5bbf" containerName="payload-processing"
Oct 31 02:05:54.879918 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:54.879860    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="59c4ccc1-c5bd-4666-a461-7802d5773296" containerName="payload-processing"
Oct 31 02:05:54.885597 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:54.885570    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:54.895455 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:54.895420    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"]
Oct 31 02:05:54.899088 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:54.899056    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-routing-overlay-volume\") pod \"4caa846a-c5cf-467c-a0d7-8889e60c5bbf\" (UID: \"4caa846a-c5cf-467c-a0d7-8889e60c5bbf\") "
Oct 31 02:05:54.899270 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:54.899231    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-plugins-config-volume\") pod \"4caa846a-c5cf-467c-a0d7-8889e60c5bbf\" (UID: \"4caa846a-c5cf-467c-a0d7-8889e60c5bbf\") "
Oct 31 02:05:54.899348 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:54.899283    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-kube-api-access-t6bcb\" (UniqueName: \"kubernetes.io/projected/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-kube-api-access-t6bcb\") pod \"4caa846a-c5cf-467c-a0d7-8889e60c5bbf\" (UID: \"4caa846a-c5cf-467c-a0d7-8889e60c5bbf\") "
Oct 31 02:05:54.899708 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:54.899659    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-provider-credentials\") pod \"4caa846a-c5cf-467c-a0d7-8889e60c5bbf\" (UID: \"4caa846a-c5cf-467c-a0d7-8889e60c5bbf\") "
Oct 31 02:05:54.900593 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:54.899674    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-routing-overlay-volume" pod "4caa846a-c5cf-467c-a0d7-8889e60c5bbf" (UID: "4caa846a-c5cf-467c-a0d7-8889e60c5bbf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:54.900702 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:54.900123    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-plugins-config-volume" pod "4caa846a-c5cf-467c-a0d7-8889e60c5bbf" (UID: "4caa846a-c5cf-467c-a0d7-8889e60c5bbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:54.903946 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:54.903921    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-provider-credentials" pod "4caa846a-c5cf-467c-a0d7-8889e60c5bbf" (UID: "4caa846a-c5cf-467c-a0d7-8889e60c5bbf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:54.904040 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:54.903969    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-kube-api-access-t6bcb" pod "4caa846a-c5cf-467c-a0d7-8889e60c5bbf" (UID: "4caa846a-c5cf-467c-a0d7-8889e60c5bbf"). InnerVolumeSpecName "kube-api-access-t6bcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:55.002023 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.001980    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-workload-socket\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.002242 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.002152    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c1e7ce-8191-44fd-a838-b9718752488a-istiod-ca-cert\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.002242 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.002223    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-podinfo\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.002358 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.002273    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-workload-certs\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.002358 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.002290    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-token\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.002492 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.002410    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pgt2\" (UniqueName: \"kubernetes.io/projected/e9c1e7ce-8191-44fd-a838-b9718752488a-kube-api-access-7pgt2\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.002492 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.002434    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-data\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.002621 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.002605    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-credential-socket\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.002679 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.002645    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-envoy\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.002961 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.002936    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6bcb\" (UniqueName: \"kubernetes.io/projected/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-kube-api-access-t6bcb\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:55.003024 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.002971    5650 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-provider-credentials\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:55.003024 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.002986    5650 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-routing-overlay-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:55.003024 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.003001    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4caa846a-c5cf-467c-a0d7-8889e60c5bbf-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:05:55.104349 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.104252    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-workload-socket\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.104534 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.104359    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c1e7ce-8191-44fd-a838-b9718752488a-istiod-ca-cert\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.104534 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.104405    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-podinfo\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.104640 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.104557    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-workload-certs\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.104640 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.104602    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-token\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.104752 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.104718    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pgt2\" (UniqueName: \"kubernetes.io/projected/e9c1e7ce-8191-44fd-a838-b9718752488a-kube-api-access-7pgt2\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.104813 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.104756    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-data\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.104813 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.104762    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-workload-socket\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.104925 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.104870    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-credential-socket\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.104986 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.104930    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-envoy\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.105730 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.105671    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-workload-certs\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.105873 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.105741    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-data\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.105873 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.105745    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-credential-socket\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.105991 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.105866    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c1e7ce-8191-44fd-a838-b9718752488a-istiod-ca-cert\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.107325 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.107306    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-podinfo\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.107613 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.107594    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-envoy\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.117817 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.117792    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-token\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.117947 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.117866    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pgt2\" (UniqueName: \"kubernetes.io/projected/e9c1e7ce-8191-44fd-a838-b9718752488a-kube-api-access-7pgt2\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.254037 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.253732    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:55.298586 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.298551    5650 generic.go:356] "Generic (PLEG): container finished" podID="4caa846a-c5cf-467c-a0d7-8889e60c5bbf" containerID="78ea270dd1625606cdca7385e73b1aed91c42ed20f4b2efbd6ed99f4776ab238" exitCode=0
Oct 31 02:05:55.298788 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.298605    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj"
Oct 31 02:05:55.298788 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.298644    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj" event={"ID":"4caa846a-c5cf-467c-a0d7-8889e60c5bbf","Type":"ContainerDied","Data":"78ea270dd1625606cdca7385e73b1aed91c42ed20f4b2efbd6ed99f4776ab238"}
Oct 31 02:05:55.298788 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.298712    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj" event={"ID":"4caa846a-c5cf-467c-a0d7-8889e60c5bbf","Type":"ContainerDied","Data":"2f7a754b3e25d31a0abe71020cb20eb6e46d19660a5000248a7785c5c05c32a1"}
Oct 31 02:05:55.298788 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.298745    5650 scope.go:122] "RemoveContainer" containerID="78ea270dd1625606cdca7385e73b1aed91c42ed20f4b2efbd6ed99f4776ab238"
Oct 31 02:05:55.312358 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.312326    5650 scope.go:122] "RemoveContainer" containerID="78ea270dd1625606cdca7385e73b1aed91c42ed20f4b2efbd6ed99f4776ab238"
Oct 31 02:05:55.312841 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:05:55.312785    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ea270dd1625606cdca7385e73b1aed91c42ed20f4b2efbd6ed99f4776ab238\": container with ID starting with 78ea270dd1625606cdca7385e73b1aed91c42ed20f4b2efbd6ed99f4776ab238 not found: ID does not exist" containerID="78ea270dd1625606cdca7385e73b1aed91c42ed20f4b2efbd6ed99f4776ab238"
Oct 31 02:05:55.312841 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.312820    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ea270dd1625606cdca7385e73b1aed91c42ed20f4b2efbd6ed99f4776ab238"} err="failed to get container status \"78ea270dd1625606cdca7385e73b1aed91c42ed20f4b2efbd6ed99f4776ab238\": rpc error: code = NotFound desc = could not find container \"78ea270dd1625606cdca7385e73b1aed91c42ed20f4b2efbd6ed99f4776ab238\": container with ID starting with 78ea270dd1625606cdca7385e73b1aed91c42ed20f4b2efbd6ed99f4776ab238 not found: ID does not exist"
Oct 31 02:05:55.336196 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.336169    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj"]
Oct 31 02:05:55.343314 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.343015    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-df9d42/payload-processing-external-model-e2e-extmod-b-df9d42-65b855cgj"]
Oct 31 02:05:55.568384 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:05:55.568348    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c1e7ce_8191_44fd_a838_b9718752488a.slice/crio-92eab140fa1cfd50f3c2bf7de2f5395ccffa46f4ef49049d53e02638bbc8a7da WatchSource:0}: Error finding container 92eab140fa1cfd50f3c2bf7de2f5395ccffa46f4ef49049d53e02638bbc8a7da: Status 404 returned error can't find the container with id 92eab140fa1cfd50f3c2bf7de2f5395ccffa46f4ef49049d53e02638bbc8a7da
Oct 31 02:05:55.570316 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.570291    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"]
Oct 31 02:05:55.572030 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.572001    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:05:55.572110 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.572063    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:05:55.572110 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:55.572091    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:05:56.199540 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:56.197243    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4caa846a-c5cf-467c-a0d7-8889e60c5bbf" path="/var/lib/kubelet/pods/4caa846a-c5cf-467c-a0d7-8889e60c5bbf/volumes"
Oct 31 02:05:56.324437 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:56.324399    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr" event={"ID":"e9c1e7ce-8191-44fd-a838-b9718752488a","Type":"ContainerStarted","Data":"2f5e6715f052080d3c5c0d7449e23e6ff65587e0ce755b1785ad3e9ca0e0f339"}
Oct 31 02:05:56.326309 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:56.326279    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr" event={"ID":"e9c1e7ce-8191-44fd-a838-b9718752488a","Type":"ContainerStarted","Data":"92eab140fa1cfd50f3c2bf7de2f5395ccffa46f4ef49049d53e02638bbc8a7da"}
Oct 31 02:05:56.362787 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:56.362737    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr" podStartSLOduration=17.362719195 podStartE2EDuration="17.362719195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:05:56.353173655 +0000 UTC m=+2712.356887947" watchObservedRunningTime="2026-10-31 02:05:56.362719195 +0000 UTC m=+2712.366433486"
Oct 31 02:05:57.254805 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:57.254768    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:05:57.256574 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:57.256546    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 31 02:05:57.256720 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:57.256614    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr" podUID="e9c1e7ce-8191-44fd-a838-b9718752488a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 31 02:05:58.255218 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:58.255183    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 31 02:05:58.255800 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:58.255242    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr" podUID="e9c1e7ce-8191-44fd-a838-b9718752488a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 31 02:05:58.555151 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:05:58.555060    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:05:59.254689 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:59.254650    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 31 02:05:59.254864 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:05:59.254717    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr" podUID="e9c1e7ce-8191-44fd-a838-b9718752488a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 31 02:06:00.254386 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:06:00.254354    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 31 02:06:00.254987 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:06:00.254420    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr" podUID="e9c1e7ce-8191-44fd-a838-b9718752488a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 31 02:06:01.254980 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:06:01.254945    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 31 02:06:01.255353 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:06:01.255003    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr" podUID="e9c1e7ce-8191-44fd-a838-b9718752488a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 31 02:06:02.256008 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:06:02.255967    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 31 02:06:02.256658 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:06:02.256611    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr" podUID="e9c1e7ce-8191-44fd-a838-b9718752488a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 31 02:06:03.259892 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:06:03.259856    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:06:03.260364 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:06:03.260347    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:06:03.261110 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:06:03.261089    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:06:28.718313 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:06:28.715202    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:06:58.557173 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:06:58.557130    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:07:26.612886 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:26.612807    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c9acb54-qfbsh"]
Oct 31 02:07:26.620585 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:26.620551    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9acb54-qfbsh"
Oct 31 02:07:26.629230 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:26.629197    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:07:26.629582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:26.629561    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:07:26.640405 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:26.640371    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c9acb54-qfbsh"]
Oct 31 02:07:26.818086 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:26.818043    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpmsd\" (UniqueName: \"kubernetes.io/projected/093087c0-0ee3-43b6-8bb7-d81f24fc7d72-kube-api-access-rpmsd\") pod \"maas-api-revoke-sub-5c9acb54-qfbsh\" (UID: \"093087c0-0ee3-43b6-8bb7-d81f24fc7d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9acb54-qfbsh"
Oct 31 02:07:26.818279 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:26.818205    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093087c0-0ee3-43b6-8bb7-d81f24fc7d72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c9acb54-qfbsh\" (UID: \"093087c0-0ee3-43b6-8bb7-d81f24fc7d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9acb54-qfbsh"
Oct 31 02:07:26.818397 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:26.818371    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093087c0-0ee3-43b6-8bb7-d81f24fc7d72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c9acb54-qfbsh\" (UID: \"093087c0-0ee3-43b6-8bb7-d81f24fc7d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9acb54-qfbsh"
Oct 31 02:07:26.920417 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:26.920308    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093087c0-0ee3-43b6-8bb7-d81f24fc7d72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c9acb54-qfbsh\" (UID: \"093087c0-0ee3-43b6-8bb7-d81f24fc7d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9acb54-qfbsh"
Oct 31 02:07:26.920417 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:26.920417    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093087c0-0ee3-43b6-8bb7-d81f24fc7d72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c9acb54-qfbsh\" (UID: \"093087c0-0ee3-43b6-8bb7-d81f24fc7d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9acb54-qfbsh"
Oct 31 02:07:26.920701 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:26.920686    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpmsd\" (UniqueName: \"kubernetes.io/projected/093087c0-0ee3-43b6-8bb7-d81f24fc7d72-kube-api-access-rpmsd\") pod \"maas-api-revoke-sub-5c9acb54-qfbsh\" (UID: \"093087c0-0ee3-43b6-8bb7-d81f24fc7d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9acb54-qfbsh"
Oct 31 02:07:26.921150 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:26.921128    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093087c0-0ee3-43b6-8bb7-d81f24fc7d72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c9acb54-qfbsh\" (UID: \"093087c0-0ee3-43b6-8bb7-d81f24fc7d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9acb54-qfbsh"
Oct 31 02:07:26.922921 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:26.922896    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093087c0-0ee3-43b6-8bb7-d81f24fc7d72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c9acb54-qfbsh\" (UID: \"093087c0-0ee3-43b6-8bb7-d81f24fc7d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9acb54-qfbsh"
Oct 31 02:07:26.934038 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:26.934008    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpmsd\" (UniqueName: \"kubernetes.io/projected/093087c0-0ee3-43b6-8bb7-d81f24fc7d72-kube-api-access-rpmsd\") pod \"maas-api-revoke-sub-5c9acb54-qfbsh\" (UID: \"093087c0-0ee3-43b6-8bb7-d81f24fc7d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9acb54-qfbsh"
Oct 31 02:07:26.949391 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:26.949361    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9acb54-qfbsh"
Oct 31 02:07:27.254108 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:27.254067    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c9acb54-qfbsh"]
Oct 31 02:07:27.257489 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:07:27.257419    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093087c0_0ee3_43b6_8bb7_d81f24fc7d72.slice/crio-630d5c3310244ab63649fdfada5f77fb54c5b5b7b725a9f3075bdda2ab26f0e8 WatchSource:0}: Error finding container 630d5c3310244ab63649fdfada5f77fb54c5b5b7b725a9f3075bdda2ab26f0e8: Status 404 returned error can't find the container with id 630d5c3310244ab63649fdfada5f77fb54c5b5b7b725a9f3075bdda2ab26f0e8
Oct 31 02:07:27.848256 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:27.848209    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9acb54-qfbsh" event={"ID":"093087c0-0ee3-43b6-8bb7-d81f24fc7d72","Type":"ContainerStarted","Data":"d00107c2f550afb50012b71ef3b7f3403eec9df505d2b493e3c1ac3df39626f5"}
Oct 31 02:07:27.848780 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:27.848271    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9acb54-qfbsh" event={"ID":"093087c0-0ee3-43b6-8bb7-d81f24fc7d72","Type":"ContainerStarted","Data":"630d5c3310244ab63649fdfada5f77fb54c5b5b7b725a9f3075bdda2ab26f0e8"}
Oct 31 02:07:27.870009 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:27.869897    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9acb54-qfbsh" podStartSLOduration=1.869881283 podStartE2EDuration="1.869881283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:07:27.868767161 +0000 UTC m=+2803.872481458" watchObservedRunningTime="2026-10-31 02:07:27.869881283 +0000 UTC m=+2803.873595574"
Oct 31 02:07:28.558019 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:07:28.557978    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:07:28.876842 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:28.876745    5650 generic.go:356] "Generic (PLEG): container finished" podID="093087c0-0ee3-43b6-8bb7-d81f24fc7d72" containerID="d00107c2f550afb50012b71ef3b7f3403eec9df505d2b493e3c1ac3df39626f5" exitCode=0
Oct 31 02:07:28.877278 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:28.877119    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9acb54-qfbsh" event={"ID":"093087c0-0ee3-43b6-8bb7-d81f24fc7d72","Type":"ContainerDied","Data":"d00107c2f550afb50012b71ef3b7f3403eec9df505d2b493e3c1ac3df39626f5"}
Oct 31 02:07:30.094206 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:30.094177    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9acb54-qfbsh"
Oct 31 02:07:30.207616 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:30.207578    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/093087c0-0ee3-43b6-8bb7-d81f24fc7d72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093087c0-0ee3-43b6-8bb7-d81f24fc7d72-maas-api-openshift-ca\") pod \"093087c0-0ee3-43b6-8bb7-d81f24fc7d72\" (UID: \"093087c0-0ee3-43b6-8bb7-d81f24fc7d72\") "
Oct 31 02:07:30.207825 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:30.207749    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/093087c0-0ee3-43b6-8bb7-d81f24fc7d72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093087c0-0ee3-43b6-8bb7-d81f24fc7d72-maas-api-xks-ca\") pod \"093087c0-0ee3-43b6-8bb7-d81f24fc7d72\" (UID: \"093087c0-0ee3-43b6-8bb7-d81f24fc7d72\") "
Oct 31 02:07:30.208143 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:30.208106    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/093087c0-0ee3-43b6-8bb7-d81f24fc7d72-maas-api-openshift-ca" pod "093087c0-0ee3-43b6-8bb7-d81f24fc7d72" (UID: "093087c0-0ee3-43b6-8bb7-d81f24fc7d72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:30.208858 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:30.208830    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093087c0-0ee3-43b6-8bb7-d81f24fc7d72-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:30.211718 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:30.211691    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/093087c0-0ee3-43b6-8bb7-d81f24fc7d72-maas-api-xks-ca" pod "093087c0-0ee3-43b6-8bb7-d81f24fc7d72" (UID: "093087c0-0ee3-43b6-8bb7-d81f24fc7d72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:30.310475 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:30.310432    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/093087c0-0ee3-43b6-8bb7-d81f24fc7d72-kube-api-access-rpmsd\" (UniqueName: \"kubernetes.io/projected/093087c0-0ee3-43b6-8bb7-d81f24fc7d72-kube-api-access-rpmsd\") pod \"093087c0-0ee3-43b6-8bb7-d81f24fc7d72\" (UID: \"093087c0-0ee3-43b6-8bb7-d81f24fc7d72\") "
Oct 31 02:07:30.311314 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:30.311283    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093087c0-0ee3-43b6-8bb7-d81f24fc7d72-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:30.312797 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:30.312764    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093087c0-0ee3-43b6-8bb7-d81f24fc7d72-kube-api-access-rpmsd" pod "093087c0-0ee3-43b6-8bb7-d81f24fc7d72" (UID: "093087c0-0ee3-43b6-8bb7-d81f24fc7d72"). InnerVolumeSpecName "kube-api-access-rpmsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:30.412701 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:30.412661    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpmsd\" (UniqueName: \"kubernetes.io/projected/093087c0-0ee3-43b6-8bb7-d81f24fc7d72-kube-api-access-rpmsd\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:30.900755 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:30.900724    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9acb54-qfbsh"
Oct 31 02:07:30.900951 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:30.900756    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9acb54-qfbsh" event={"ID":"093087c0-0ee3-43b6-8bb7-d81f24fc7d72","Type":"ContainerDied","Data":"630d5c3310244ab63649fdfada5f77fb54c5b5b7b725a9f3075bdda2ab26f0e8"}
Oct 31 02:07:30.900951 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:30.900818    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="630d5c3310244ab63649fdfada5f77fb54c5b5b7b725a9f3075bdda2ab26f0e8"
Oct 31 02:07:38.844648 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:38.844609    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg"]
Oct 31 02:07:43.846130 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:43.846044    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg" podUID="bb268bae-9d2f-493a-81e5-fdc55ebd9b11" containerName="payload-processing" containerID="cri-o://4d72f2c3278518da5912ab8bad71e06aaeb4e8eaf6970b42e5792ba503039f6a" gracePeriod=25
Oct 31 02:07:44.181071 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.181037    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg"
Oct 31 02:07:44.230098 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.230057    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6"]
Oct 31 02:07:44.231607 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.231575    5650 cpu_manager.go:414] "containerMap: removing container" podUID="bb268bae-9d2f-493a-81e5-fdc55ebd9b11" containerName="payload-processing"
Oct 31 02:07:44.231607 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.231605    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb268bae-9d2f-493a-81e5-fdc55ebd9b11" containerName="payload-processing"
Oct 31 02:07:44.231838 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.231733    5650 cpu_manager.go:414] "containerMap: removing container" podUID="093087c0-0ee3-43b6-8bb7-d81f24fc7d72" containerName="revoke-keys"
Oct 31 02:07:44.231838 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.231743    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093087c0-0ee3-43b6-8bb7-d81f24fc7d72" containerName="revoke-keys"
Oct 31 02:07:44.231997 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.231977    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="093087c0-0ee3-43b6-8bb7-d81f24fc7d72" containerName="revoke-keys"
Oct 31 02:07:44.232066 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.232009    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb268bae-9d2f-493a-81e5-fdc55ebd9b11" containerName="payload-processing"
Oct 31 02:07:44.238264 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.238231    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6"
Oct 31 02:07:44.239041 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.238965    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6"]
Oct 31 02:07:44.243897 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.243837    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bb5c51\"/\"payload-processing-external-model-plugins-e2e-extmod-a-bb5c51\""
Oct 31 02:07:44.244048 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.243937    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bb5c51\"/\"agc-cred-e107317a3c62\""
Oct 31 02:07:44.244493 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.244430    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bb5c51\"/\"routing-overlay\""
Oct 31 02:07:44.244607 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.244548    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bb5c51\"/\"kube-root-ca.crt\""
Oct 31 02:07:44.244674 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.244609    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bb5c51\"/\"openshift-service-ca.crt\""
Oct 31 02:07:44.244807 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.244787    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bb5c51\"/\"payload-processing-external-model-e2e--43356fff-dockercfg-qvkjn\""
Oct 31 02:07:44.250007 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.249980    5650 generic.go:356] "Generic (PLEG): container finished" podID="bb268bae-9d2f-493a-81e5-fdc55ebd9b11" containerID="4d72f2c3278518da5912ab8bad71e06aaeb4e8eaf6970b42e5792ba503039f6a" exitCode=0
Oct 31 02:07:44.250127 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.250023    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg"
Oct 31 02:07:44.250127 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.250065    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg" event={"ID":"bb268bae-9d2f-493a-81e5-fdc55ebd9b11","Type":"ContainerDied","Data":"4d72f2c3278518da5912ab8bad71e06aaeb4e8eaf6970b42e5792ba503039f6a"}
Oct 31 02:07:44.250127 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.250117    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg" event={"ID":"bb268bae-9d2f-493a-81e5-fdc55ebd9b11","Type":"ContainerDied","Data":"f162655181a961fb94b20bea279c7c6b3aa600f128c1c631a196870a93dbd457"}
Oct 31 02:07:44.250287 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.250148    5650 scope.go:122] "RemoveContainer" containerID="4d72f2c3278518da5912ab8bad71e06aaeb4e8eaf6970b42e5792ba503039f6a"
Oct 31 02:07:44.267324 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.267297    5650 scope.go:122] "RemoveContainer" containerID="4d72f2c3278518da5912ab8bad71e06aaeb4e8eaf6970b42e5792ba503039f6a"
Oct 31 02:07:44.267747 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:07:44.267724    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d72f2c3278518da5912ab8bad71e06aaeb4e8eaf6970b42e5792ba503039f6a\": container with ID starting with 4d72f2c3278518da5912ab8bad71e06aaeb4e8eaf6970b42e5792ba503039f6a not found: ID does not exist" containerID="4d72f2c3278518da5912ab8bad71e06aaeb4e8eaf6970b42e5792ba503039f6a"
Oct 31 02:07:44.267850 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.267759    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d72f2c3278518da5912ab8bad71e06aaeb4e8eaf6970b42e5792ba503039f6a"} err="failed to get container status \"4d72f2c3278518da5912ab8bad71e06aaeb4e8eaf6970b42e5792ba503039f6a\": rpc error: code = NotFound desc = could not find container \"4d72f2c3278518da5912ab8bad71e06aaeb4e8eaf6970b42e5792ba503039f6a\": container with ID starting with 4d72f2c3278518da5912ab8bad71e06aaeb4e8eaf6970b42e5792ba503039f6a not found: ID does not exist"
Oct 31 02:07:44.284835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.284789    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-plugins-config-volume\") pod \"bb268bae-9d2f-493a-81e5-fdc55ebd9b11\" (UID: \"bb268bae-9d2f-493a-81e5-fdc55ebd9b11\") "
Oct 31 02:07:44.284995 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.284872    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-provider-credentials\") pod \"bb268bae-9d2f-493a-81e5-fdc55ebd9b11\" (UID: \"bb268bae-9d2f-493a-81e5-fdc55ebd9b11\") "
Oct 31 02:07:44.284995 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.284946    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-routing-overlay-volume\") pod \"bb268bae-9d2f-493a-81e5-fdc55ebd9b11\" (UID: \"bb268bae-9d2f-493a-81e5-fdc55ebd9b11\") "
Oct 31 02:07:44.285185 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.285162    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-kube-api-access-ts5xp\" (UniqueName: \"kubernetes.io/projected/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-kube-api-access-ts5xp\") pod \"bb268bae-9d2f-493a-81e5-fdc55ebd9b11\" (UID: \"bb268bae-9d2f-493a-81e5-fdc55ebd9b11\") "
Oct 31 02:07:44.285313 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.285284    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-plugins-config-volume" pod "bb268bae-9d2f-493a-81e5-fdc55ebd9b11" (UID: "bb268bae-9d2f-493a-81e5-fdc55ebd9b11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:44.285711 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.285687    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-routing-overlay-volume" pod "bb268bae-9d2f-493a-81e5-fdc55ebd9b11" (UID: "bb268bae-9d2f-493a-81e5-fdc55ebd9b11"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:44.285875 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.285712    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a38f57-8c85-4138-8853-78cac83398cb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6\" (UID: \"19a38f57-8c85-4138-8853-78cac83398cb\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6"
Oct 31 02:07:44.286027 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.285999    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9gp5\" (UniqueName: \"kubernetes.io/projected/19a38f57-8c85-4138-8853-78cac83398cb-kube-api-access-l9gp5\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6\" (UID: \"19a38f57-8c85-4138-8853-78cac83398cb\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6"
Oct 31 02:07:44.286242 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.286216    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19a38f57-8c85-4138-8853-78cac83398cb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6\" (UID: \"19a38f57-8c85-4138-8853-78cac83398cb\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6"
Oct 31 02:07:44.286458 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.286411    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19a38f57-8c85-4138-8853-78cac83398cb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6\" (UID: \"19a38f57-8c85-4138-8853-78cac83398cb\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6"
Oct 31 02:07:44.286822 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.286795    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.286942 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.286824    5650 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-routing-overlay-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.287527 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.287504    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-provider-credentials" pod "bb268bae-9d2f-493a-81e5-fdc55ebd9b11" (UID: "bb268bae-9d2f-493a-81e5-fdc55ebd9b11"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:44.288386 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.288368    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-kube-api-access-ts5xp" pod "bb268bae-9d2f-493a-81e5-fdc55ebd9b11" (UID: "bb268bae-9d2f-493a-81e5-fdc55ebd9b11"). InnerVolumeSpecName "kube-api-access-ts5xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:44.388774 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.388681    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9gp5\" (UniqueName: \"kubernetes.io/projected/19a38f57-8c85-4138-8853-78cac83398cb-kube-api-access-l9gp5\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6\" (UID: \"19a38f57-8c85-4138-8853-78cac83398cb\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6"
Oct 31 02:07:44.388935 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.388819    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19a38f57-8c85-4138-8853-78cac83398cb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6\" (UID: \"19a38f57-8c85-4138-8853-78cac83398cb\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6"
Oct 31 02:07:44.388935 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.388919    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19a38f57-8c85-4138-8853-78cac83398cb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6\" (UID: \"19a38f57-8c85-4138-8853-78cac83398cb\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6"
Oct 31 02:07:44.389157 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.389139    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a38f57-8c85-4138-8853-78cac83398cb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6\" (UID: \"19a38f57-8c85-4138-8853-78cac83398cb\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6"
Oct 31 02:07:44.389320 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.389307    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts5xp\" (UniqueName: \"kubernetes.io/projected/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-kube-api-access-ts5xp\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.389363 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.389323    5650 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb268bae-9d2f-493a-81e5-fdc55ebd9b11-provider-credentials\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.390110 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.390081    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19a38f57-8c85-4138-8853-78cac83398cb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6\" (UID: \"19a38f57-8c85-4138-8853-78cac83398cb\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6"
Oct 31 02:07:44.390299 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.390279    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a38f57-8c85-4138-8853-78cac83398cb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6\" (UID: \"19a38f57-8c85-4138-8853-78cac83398cb\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6"
Oct 31 02:07:44.391970 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.391942    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19a38f57-8c85-4138-8853-78cac83398cb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6\" (UID: \"19a38f57-8c85-4138-8853-78cac83398cb\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6"
Oct 31 02:07:44.402223 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.402196    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9gp5\" (UniqueName: \"kubernetes.io/projected/19a38f57-8c85-4138-8853-78cac83398cb-kube-api-access-l9gp5\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6\" (UID: \"19a38f57-8c85-4138-8853-78cac83398cb\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6"
Oct 31 02:07:44.579165 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.579127    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6"
Oct 31 02:07:44.612515 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.612482    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg"]
Oct 31 02:07:44.619062 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.619028    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-65467rqtg"]
Oct 31 02:07:44.882136 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:07:44.882107    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a38f57_8c85_4138_8853_78cac83398cb.slice/crio-2e9e0bf1dd2f06eb6bc71082bd64fdd0ca17a86186cb1f7e193ec485995f5b4d WatchSource:0}: Error finding container 2e9e0bf1dd2f06eb6bc71082bd64fdd0ca17a86186cb1f7e193ec485995f5b4d: Status 404 returned error can't find the container with id 2e9e0bf1dd2f06eb6bc71082bd64fdd0ca17a86186cb1f7e193ec485995f5b4d
Oct 31 02:07:44.883337 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:44.883305    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6"]
Oct 31 02:07:45.298970 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:45.298909    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6" event={"ID":"19a38f57-8c85-4138-8853-78cac83398cb","Type":"ContainerStarted","Data":"8528196cfcb1513db8971ee84e04e0bdfe75f11832a35a2e08b8f1d653c1b2cc"}
Oct 31 02:07:45.299281 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:45.299255    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6" event={"ID":"19a38f57-8c85-4138-8853-78cac83398cb","Type":"ContainerStarted","Data":"2e9e0bf1dd2f06eb6bc71082bd64fdd0ca17a86186cb1f7e193ec485995f5b4d"}
Oct 31 02:07:45.300804 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:45.300781    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6"
Oct 31 02:07:45.333636 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:45.333586    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6" podStartSLOduration=11.33357037 podStartE2EDuration="11.33357037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:07:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:07:45.328313343 +0000 UTC m=+2821.332027635" watchObservedRunningTime="2026-10-31 02:07:45.33357037 +0000 UTC m=+2821.337284699"
Oct 31 02:07:46.122945 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:46.122895    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb268bae-9d2f-493a-81e5-fdc55ebd9b11" path="/var/lib/kubelet/pods/bb268bae-9d2f-493a-81e5-fdc55ebd9b11/volumes"
Oct 31 02:07:46.957921 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:46.957882    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6"]
Oct 31 02:07:52.379761 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.359169    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6" podUID="19a38f57-8c85-4138-8853-78cac83398cb" containerName="payload-processing" containerID="cri-o://8528196cfcb1513db8971ee84e04e0bdfe75f11832a35a2e08b8f1d653c1b2cc" gracePeriod=25
Oct 31 02:07:52.543607 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:07:52.543560    5650 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a38f57_8c85_4138_8853_78cac83398cb.slice/crio-conmon-8528196cfcb1513db8971ee84e04e0bdfe75f11832a35a2e08b8f1d653c1b2cc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a38f57_8c85_4138_8853_78cac83398cb.slice/crio-8528196cfcb1513db8971ee84e04e0bdfe75f11832a35a2e08b8f1d653c1b2cc.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 02:07:52.644518 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.644228    5650 generic.go:356] "Generic (PLEG): container finished" podID="19a38f57-8c85-4138-8853-78cac83398cb" containerID="8528196cfcb1513db8971ee84e04e0bdfe75f11832a35a2e08b8f1d653c1b2cc" exitCode=0
Oct 31 02:07:52.644701 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.644537    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6" event={"ID":"19a38f57-8c85-4138-8853-78cac83398cb","Type":"ContainerDied","Data":"8528196cfcb1513db8971ee84e04e0bdfe75f11832a35a2e08b8f1d653c1b2cc"}
Oct 31 02:07:52.755829 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.755800    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6"
Oct 31 02:07:52.796421 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.796387    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8"]
Oct 31 02:07:52.797836 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.797804    5650 cpu_manager.go:414] "containerMap: removing container" podUID="19a38f57-8c85-4138-8853-78cac83398cb" containerName="payload-processing"
Oct 31 02:07:52.797836 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.797836    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a38f57-8c85-4138-8853-78cac83398cb" containerName="payload-processing"
Oct 31 02:07:52.800061 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.798124    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a38f57-8c85-4138-8853-78cac83398cb" containerName="payload-processing"
Oct 31 02:07:52.819006 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.818973    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8"]
Oct 31 02:07:52.819208 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.819101    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8"
Oct 31 02:07:52.826530 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.826484    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"routing-overlay\""
Oct 31 02:07:52.830948 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.830759    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"e2e-cfl-cec4cc-provider-cred\""
Oct 31 02:07:52.830948 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.830900    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"payload-processing-external-model-e2e--1b1bd4a9-dockercfg-7r8fh\""
Oct 31 02:07:52.831273 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.830938    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"agc-cred-e866e2693d60\""
Oct 31 02:07:52.831273 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.831158    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"openshift-service-ca.crt\""
Oct 31 02:07:52.831870 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.831848    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"kube-root-ca.crt\""
Oct 31 02:07:52.832058 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.831915    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"payload-processing-external-model-plugins-e2e-extmod-a-28e4f3\""
Oct 31 02:07:52.887533 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.887492    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19a38f57-8c85-4138-8853-78cac83398cb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19a38f57-8c85-4138-8853-78cac83398cb-routing-overlay-volume\") pod \"19a38f57-8c85-4138-8853-78cac83398cb\" (UID: \"19a38f57-8c85-4138-8853-78cac83398cb\") "
Oct 31 02:07:52.887718 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.887544    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a38f57-8c85-4138-8853-78cac83398cb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/19a38f57-8c85-4138-8853-78cac83398cb-provider-credentials\") pod \"19a38f57-8c85-4138-8853-78cac83398cb\" (UID: \"19a38f57-8c85-4138-8853-78cac83398cb\") "
Oct 31 02:07:52.887718 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.887655    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19a38f57-8c85-4138-8853-78cac83398cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a38f57-8c85-4138-8853-78cac83398cb-plugins-config-volume\") pod \"19a38f57-8c85-4138-8853-78cac83398cb\" (UID: \"19a38f57-8c85-4138-8853-78cac83398cb\") "
Oct 31 02:07:52.887861 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.887764    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a38f57-8c85-4138-8853-78cac83398cb-kube-api-access-l9gp5\" (UniqueName: \"kubernetes.io/projected/19a38f57-8c85-4138-8853-78cac83398cb-kube-api-access-l9gp5\") pod \"19a38f57-8c85-4138-8853-78cac83398cb\" (UID: \"19a38f57-8c85-4138-8853-78cac83398cb\") "
Oct 31 02:07:52.887994 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.887966    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a38f57-8c85-4138-8853-78cac83398cb-routing-overlay-volume" pod "19a38f57-8c85-4138-8853-78cac83398cb" (UID: "19a38f57-8c85-4138-8853-78cac83398cb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:52.888076 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.888054    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a38f57-8c85-4138-8853-78cac83398cb-plugins-config-volume" pod "19a38f57-8c85-4138-8853-78cac83398cb" (UID: "19a38f57-8c85-4138-8853-78cac83398cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:52.888404 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.888382    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bljwg\" (UniqueName: \"kubernetes.io/projected/d2297521-9d8a-4b65-9eb0-8aeae1899606-kube-api-access-bljwg\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8\" (UID: \"d2297521-9d8a-4b65-9eb0-8aeae1899606\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8"
Oct 31 02:07:52.888528 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.888490    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2297521-9d8a-4b65-9eb0-8aeae1899606-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8\" (UID: \"d2297521-9d8a-4b65-9eb0-8aeae1899606\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8"
Oct 31 02:07:52.888778 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.888758    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2297521-9d8a-4b65-9eb0-8aeae1899606-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8\" (UID: \"d2297521-9d8a-4b65-9eb0-8aeae1899606\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8"
Oct 31 02:07:52.888848 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.888796    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2297521-9d8a-4b65-9eb0-8aeae1899606-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8\" (UID: \"d2297521-9d8a-4b65-9eb0-8aeae1899606\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8"
Oct 31 02:07:52.889066 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.889041    5650 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19a38f57-8c85-4138-8853-78cac83398cb-routing-overlay-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:52.889132 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.889072    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a38f57-8c85-4138-8853-78cac83398cb-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:52.890250 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.890229    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a38f57-8c85-4138-8853-78cac83398cb-kube-api-access-l9gp5" pod "19a38f57-8c85-4138-8853-78cac83398cb" (UID: "19a38f57-8c85-4138-8853-78cac83398cb"). InnerVolumeSpecName "kube-api-access-l9gp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:52.890432 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.890413    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a38f57-8c85-4138-8853-78cac83398cb-provider-credentials" pod "19a38f57-8c85-4138-8853-78cac83398cb" (UID: "19a38f57-8c85-4138-8853-78cac83398cb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:52.990668 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.990626    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2297521-9d8a-4b65-9eb0-8aeae1899606-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8\" (UID: \"d2297521-9d8a-4b65-9eb0-8aeae1899606\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8"
Oct 31 02:07:52.990862 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.990697    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2297521-9d8a-4b65-9eb0-8aeae1899606-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8\" (UID: \"d2297521-9d8a-4b65-9eb0-8aeae1899606\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8"
Oct 31 02:07:52.990955 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.990882    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bljwg\" (UniqueName: \"kubernetes.io/projected/d2297521-9d8a-4b65-9eb0-8aeae1899606-kube-api-access-bljwg\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8\" (UID: \"d2297521-9d8a-4b65-9eb0-8aeae1899606\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8"
Oct 31 02:07:52.991024 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.990987    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2297521-9d8a-4b65-9eb0-8aeae1899606-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8\" (UID: \"d2297521-9d8a-4b65-9eb0-8aeae1899606\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8"
Oct 31 02:07:52.991261 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.991242    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9gp5\" (UniqueName: \"kubernetes.io/projected/19a38f57-8c85-4138-8853-78cac83398cb-kube-api-access-l9gp5\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:52.991435 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.991416    5650 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19a38f57-8c85-4138-8853-78cac83398cb-provider-credentials\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:07:52.992123 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.992092    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2297521-9d8a-4b65-9eb0-8aeae1899606-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8\" (UID: \"d2297521-9d8a-4b65-9eb0-8aeae1899606\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8"
Oct 31 02:07:52.992230 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.992128    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2297521-9d8a-4b65-9eb0-8aeae1899606-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8\" (UID: \"d2297521-9d8a-4b65-9eb0-8aeae1899606\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8"
Oct 31 02:07:52.994689 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:52.994664    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2297521-9d8a-4b65-9eb0-8aeae1899606-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8\" (UID: \"d2297521-9d8a-4b65-9eb0-8aeae1899606\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8"
Oct 31 02:07:53.005200 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:53.005168    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bljwg\" (UniqueName: \"kubernetes.io/projected/d2297521-9d8a-4b65-9eb0-8aeae1899606-kube-api-access-bljwg\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8\" (UID: \"d2297521-9d8a-4b65-9eb0-8aeae1899606\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8"
Oct 31 02:07:53.154243 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:53.153942    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8"
Oct 31 02:07:53.532527 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:53.532494    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8"]
Oct 31 02:07:53.533003 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:07:53.532971    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2297521_9d8a_4b65_9eb0_8aeae1899606.slice/crio-a0c8f31b60d8f91f4bb76169d30d71c3d8c39a6c93a67848777663f6fe8d65a8 WatchSource:0}: Error finding container a0c8f31b60d8f91f4bb76169d30d71c3d8c39a6c93a67848777663f6fe8d65a8: Status 404 returned error can't find the container with id a0c8f31b60d8f91f4bb76169d30d71c3d8c39a6c93a67848777663f6fe8d65a8
Oct 31 02:07:53.660609 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:53.660568    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6" event={"ID":"19a38f57-8c85-4138-8853-78cac83398cb","Type":"ContainerDied","Data":"2e9e0bf1dd2f06eb6bc71082bd64fdd0ca17a86186cb1f7e193ec485995f5b4d"}
Oct 31 02:07:53.660609 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:53.660599    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6"
Oct 31 02:07:53.660923 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:53.660655    5650 scope.go:122] "RemoveContainer" containerID="8528196cfcb1513db8971ee84e04e0bdfe75f11832a35a2e08b8f1d653c1b2cc"
Oct 31 02:07:53.663045 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:53.663011    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8" event={"ID":"d2297521-9d8a-4b65-9eb0-8aeae1899606","Type":"ContainerStarted","Data":"c15756e11baed0c47bab583e52812664185a4bd253599ef1630025beaf39f6b0"}
Oct 31 02:07:53.663183 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:53.663104    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8"
Oct 31 02:07:53.663183 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:53.663151    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8" event={"ID":"d2297521-9d8a-4b65-9eb0-8aeae1899606","Type":"ContainerStarted","Data":"a0c8f31b60d8f91f4bb76169d30d71c3d8c39a6c93a67848777663f6fe8d65a8"}
Oct 31 02:07:53.752818 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:53.752776    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6"]
Oct 31 02:07:53.764248 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:07:53.764212    5650 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-28e4f3/agc-cred-e866e2693d60: secret "agc-cred-e866e2693d60" not found
Oct 31 02:07:53.764248 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:07:53.764251    5650 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8: secret "agc-cred-e866e2693d60" not found
Oct 31 02:07:53.764514 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:07:53.764322    5650 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2297521-9d8a-4b65-9eb0-8aeae1899606-provider-credentials podName:d2297521-9d8a-4b65-9eb0-8aeae1899606 nodeName:}" failed. No retries permitted until 2026-10-31 02:07:54.264297508 +0000 UTC m=+2830.268011790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d2297521-9d8a-4b65-9eb0-8aeae1899606-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8" (UID: "d2297521-9d8a-4b65-9eb0-8aeae1899606") : secret "agc-cred-e866e2693d60" not found
Oct 31 02:07:53.768078 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:53.768050    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-765c8t4w6"]
Oct 31 02:07:53.851103 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:53.850988    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8" podStartSLOduration=62.850965843 podStartE2EDuration="1m2.850965843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:07:53.846724927 +0000 UTC m=+2829.850439219" watchObservedRunningTime="2026-10-31 02:07:53.850965843 +0000 UTC m=+2829.854680135"
Oct 31 02:07:54.094470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:54.090596    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8"]
Oct 31 02:07:54.171059 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:07:54.168743    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19a38f57-8c85-4138-8853-78cac83398cb" path="/var/lib/kubelet/pods/19a38f57-8c85-4138-8853-78cac83398cb/volumes"
Oct 31 02:07:54.280090 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:07:54.279945    5650 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-28e4f3/agc-cred-e866e2693d60: secret "agc-cred-e866e2693d60" not found
Oct 31 02:07:54.280090 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:07:54.279984    5650 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8: secret "agc-cred-e866e2693d60" not found
Oct 31 02:07:54.280090 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:07:54.280051    5650 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2297521-9d8a-4b65-9eb0-8aeae1899606-provider-credentials podName:d2297521-9d8a-4b65-9eb0-8aeae1899606 nodeName:}" failed. No retries permitted until 2026-10-31 02:07:55.280032009 +0000 UTC m=+2831.283746299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d2297521-9d8a-4b65-9eb0-8aeae1899606-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8" (UID: "d2297521-9d8a-4b65-9eb0-8aeae1899606") : secret "agc-cred-e866e2693d60" not found
Oct 31 02:07:55.445475 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:07:55.420939    5650 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-28e4f3/agc-cred-e866e2693d60: secret "agc-cred-e866e2693d60" not found
Oct 31 02:07:55.446220 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:07:55.446195    5650 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8: secret "agc-cred-e866e2693d60" not found
Oct 31 02:07:55.446421 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:07:55.446406    5650 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2297521-9d8a-4b65-9eb0-8aeae1899606-provider-credentials podName:d2297521-9d8a-4b65-9eb0-8aeae1899606 nodeName:}" failed. No retries permitted until 2026-10-31 02:07:57.446379554 +0000 UTC m=+2833.450093838 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d2297521-9d8a-4b65-9eb0-8aeae1899606-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8" (UID: "d2297521-9d8a-4b65-9eb0-8aeae1899606") : secret "agc-cred-e866e2693d60" not found
Oct 31 02:07:57.478684 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:07:57.478644    5650 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-28e4f3/agc-cred-e866e2693d60: secret "agc-cred-e866e2693d60" not found
Oct 31 02:07:57.478684 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:07:57.478681    5650 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8: secret "agc-cred-e866e2693d60" not found
Oct 31 02:07:57.479296 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:07:57.478766    5650 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2297521-9d8a-4b65-9eb0-8aeae1899606-provider-credentials podName:d2297521-9d8a-4b65-9eb0-8aeae1899606 nodeName:}" failed. No retries permitted until 2026-10-31 02:08:01.478744608 +0000 UTC m=+2837.482458894 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d2297521-9d8a-4b65-9eb0-8aeae1899606-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8" (UID: "d2297521-9d8a-4b65-9eb0-8aeae1899606") : secret "agc-cred-e866e2693d60" not found
Oct 31 02:07:58.561495 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:07:58.561429    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:08:00.796077 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:00.796035    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8" podUID="d2297521-9d8a-4b65-9eb0-8aeae1899606" containerName="payload-processing" containerID="cri-o://c15756e11baed0c47bab583e52812664185a4bd253599ef1630025beaf39f6b0" gracePeriod=25
Oct 31 02:08:00.936664 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:00.936599    5650 generic.go:356] "Generic (PLEG): container finished" podID="d2297521-9d8a-4b65-9eb0-8aeae1899606" containerID="c15756e11baed0c47bab583e52812664185a4bd253599ef1630025beaf39f6b0" exitCode=0
Oct 31 02:08:00.936852 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:00.936672    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8" event={"ID":"d2297521-9d8a-4b65-9eb0-8aeae1899606","Type":"ContainerDied","Data":"c15756e11baed0c47bab583e52812664185a4bd253599ef1630025beaf39f6b0"}
Oct 31 02:08:01.166843 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.166817    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8"
Oct 31 02:08:01.206299 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.206265    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-8644d884b7-fgzql"]
Oct 31 02:08:01.207665 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.207633    5650 cpu_manager.go:414] "containerMap: removing container" podUID="d2297521-9d8a-4b65-9eb0-8aeae1899606" containerName="payload-processing"
Oct 31 02:08:01.207665 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.207659    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2297521-9d8a-4b65-9eb0-8aeae1899606" containerName="payload-processing"
Oct 31 02:08:01.207921 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.207905    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2297521-9d8a-4b65-9eb0-8aeae1899606" containerName="payload-processing"
Oct 31 02:08:01.212833 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.212799    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2297521-9d8a-4b65-9eb0-8aeae1899606-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2297521-9d8a-4b65-9eb0-8aeae1899606-plugins-config-volume\") pod \"d2297521-9d8a-4b65-9eb0-8aeae1899606\" (UID: \"d2297521-9d8a-4b65-9eb0-8aeae1899606\") "
Oct 31 02:08:01.213017 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.212970    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2297521-9d8a-4b65-9eb0-8aeae1899606-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2297521-9d8a-4b65-9eb0-8aeae1899606-provider-credentials\") pod \"d2297521-9d8a-4b65-9eb0-8aeae1899606\" (UID: \"d2297521-9d8a-4b65-9eb0-8aeae1899606\") "
Oct 31 02:08:01.213195 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.213174    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2297521-9d8a-4b65-9eb0-8aeae1899606-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2297521-9d8a-4b65-9eb0-8aeae1899606-routing-overlay-volume\") pod \"d2297521-9d8a-4b65-9eb0-8aeae1899606\" (UID: \"d2297521-9d8a-4b65-9eb0-8aeae1899606\") "
Oct 31 02:08:01.213305 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.213283    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2297521-9d8a-4b65-9eb0-8aeae1899606-kube-api-access-bljwg\" (UniqueName: \"kubernetes.io/projected/d2297521-9d8a-4b65-9eb0-8aeae1899606-kube-api-access-bljwg\") pod \"d2297521-9d8a-4b65-9eb0-8aeae1899606\" (UID: \"d2297521-9d8a-4b65-9eb0-8aeae1899606\") "
Oct 31 02:08:01.215828 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.215672    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8644d884b7-fgzql"
Oct 31 02:08:01.216897 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.216782    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2297521-9d8a-4b65-9eb0-8aeae1899606-plugins-config-volume" pod "d2297521-9d8a-4b65-9eb0-8aeae1899606" (UID: "d2297521-9d8a-4b65-9eb0-8aeae1899606"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:08:01.218689 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.218609    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2297521-9d8a-4b65-9eb0-8aeae1899606-routing-overlay-volume" pod "d2297521-9d8a-4b65-9eb0-8aeae1899606" (UID: "d2297521-9d8a-4b65-9eb0-8aeae1899606"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:08:01.221384 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.221360    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2297521-9d8a-4b65-9eb0-8aeae1899606-kube-api-access-bljwg" pod "d2297521-9d8a-4b65-9eb0-8aeae1899606" (UID: "d2297521-9d8a-4b65-9eb0-8aeae1899606"). InnerVolumeSpecName "kube-api-access-bljwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:01.221514 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.221414    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2297521-9d8a-4b65-9eb0-8aeae1899606-provider-credentials" pod "d2297521-9d8a-4b65-9eb0-8aeae1899606" (UID: "d2297521-9d8a-4b65-9eb0-8aeae1899606"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:01.222843 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.222820    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 02:08:01.222955 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.222899    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 02:08:01.223020 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.222966    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-e1dd0b729916\""
Oct 31 02:08:01.223089 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.223027    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-6e1bafd9570a\""
Oct 31 02:08:01.223151 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.223089    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 02:08:01.223151 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.223144    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 02:08:01.223277 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.223261    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-qsc4d\""
Oct 31 02:08:01.242189 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.242153    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8644d884b7-fgzql"]
Oct 31 02:08:01.317081 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.317043    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-plugins-config-volume\") pod \"payload-processing-external-model-8644d884b7-fgzql\" (UID: \"19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec\") " pod="models-as-a-service/payload-processing-external-model-8644d884b7-fgzql"
Oct 31 02:08:01.317314 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.317201    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-provider-credentials\") pod \"payload-processing-external-model-8644d884b7-fgzql\" (UID: \"19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec\") " pod="models-as-a-service/payload-processing-external-model-8644d884b7-fgzql"
Oct 31 02:08:01.317386 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.317338    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-routing-overlay-volume\") pod \"payload-processing-external-model-8644d884b7-fgzql\" (UID: \"19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec\") " pod="models-as-a-service/payload-processing-external-model-8644d884b7-fgzql"
Oct 31 02:08:01.317749 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.317726    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwpq6\" (UniqueName: \"kubernetes.io/projected/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-kube-api-access-fwpq6\") pod \"payload-processing-external-model-8644d884b7-fgzql\" (UID: \"19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec\") " pod="models-as-a-service/payload-processing-external-model-8644d884b7-fgzql"
Oct 31 02:08:01.317901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.317886    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2297521-9d8a-4b65-9eb0-8aeae1899606-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:08:01.317957 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.317907    5650 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d2297521-9d8a-4b65-9eb0-8aeae1899606-provider-credentials\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:08:01.317957 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.317923    5650 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d2297521-9d8a-4b65-9eb0-8aeae1899606-routing-overlay-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:08:01.317957 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.317941    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bljwg\" (UniqueName: \"kubernetes.io/projected/d2297521-9d8a-4b65-9eb0-8aeae1899606-kube-api-access-bljwg\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:08:01.420131 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.420044    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-provider-credentials\") pod \"payload-processing-external-model-8644d884b7-fgzql\" (UID: \"19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec\") " pod="models-as-a-service/payload-processing-external-model-8644d884b7-fgzql"
Oct 31 02:08:01.420292 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.420173    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-routing-overlay-volume\") pod \"payload-processing-external-model-8644d884b7-fgzql\" (UID: \"19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec\") " pod="models-as-a-service/payload-processing-external-model-8644d884b7-fgzql"
Oct 31 02:08:01.420539 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.420494    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwpq6\" (UniqueName: \"kubernetes.io/projected/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-kube-api-access-fwpq6\") pod \"payload-processing-external-model-8644d884b7-fgzql\" (UID: \"19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec\") " pod="models-as-a-service/payload-processing-external-model-8644d884b7-fgzql"
Oct 31 02:08:01.420654 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.420604    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-plugins-config-volume\") pod \"payload-processing-external-model-8644d884b7-fgzql\" (UID: \"19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec\") " pod="models-as-a-service/payload-processing-external-model-8644d884b7-fgzql"
Oct 31 02:08:01.421044 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.421014    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-routing-overlay-volume\") pod \"payload-processing-external-model-8644d884b7-fgzql\" (UID: \"19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec\") " pod="models-as-a-service/payload-processing-external-model-8644d884b7-fgzql"
Oct 31 02:08:01.421312 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.421294    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-plugins-config-volume\") pod \"payload-processing-external-model-8644d884b7-fgzql\" (UID: \"19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec\") " pod="models-as-a-service/payload-processing-external-model-8644d884b7-fgzql"
Oct 31 02:08:01.422851 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.422830    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-provider-credentials\") pod \"payload-processing-external-model-8644d884b7-fgzql\" (UID: \"19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec\") " pod="models-as-a-service/payload-processing-external-model-8644d884b7-fgzql"
Oct 31 02:08:01.434006 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.433974    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwpq6\" (UniqueName: \"kubernetes.io/projected/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-kube-api-access-fwpq6\") pod \"payload-processing-external-model-8644d884b7-fgzql\" (UID: \"19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec\") " pod="models-as-a-service/payload-processing-external-model-8644d884b7-fgzql"
Oct 31 02:08:01.560934 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.560897    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8644d884b7-fgzql"
Oct 31 02:08:01.867025 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:08:01.866980    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f93a29_bc3a_4f14_af5a_7f33a0d5b1ec.slice/crio-012cc2df07ae4452266305997d5adcbff87d91571021dfbd51277e09b964ad56 WatchSource:0}: Error finding container 012cc2df07ae4452266305997d5adcbff87d91571021dfbd51277e09b964ad56: Status 404 returned error can't find the container with id 012cc2df07ae4452266305997d5adcbff87d91571021dfbd51277e09b964ad56
Oct 31 02:08:01.867495 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.867383    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8644d884b7-fgzql"]
Oct 31 02:08:01.960335 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.960306    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8"
Oct 31 02:08:01.960335 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.960319    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8" event={"ID":"d2297521-9d8a-4b65-9eb0-8aeae1899606","Type":"ContainerDied","Data":"a0c8f31b60d8f91f4bb76169d30d71c3d8c39a6c93a67848777663f6fe8d65a8"}
Oct 31 02:08:01.960632 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.960413    5650 scope.go:122] "RemoveContainer" containerID="c15756e11baed0c47bab583e52812664185a4bd253599ef1630025beaf39f6b0"
Oct 31 02:08:01.965437 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.965393    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8644d884b7-fgzql" event={"ID":"19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec","Type":"ContainerStarted","Data":"012cc2df07ae4452266305997d5adcbff87d91571021dfbd51277e09b964ad56"}
Oct 31 02:08:01.994701 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:01.994668    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8"]
Oct 31 02:08:02.004389 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:02.004362    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-5f4ddfdh8"]
Oct 31 02:08:02.123821 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:02.123739    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2297521-9d8a-4b65-9eb0-8aeae1899606" path="/var/lib/kubelet/pods/d2297521-9d8a-4b65-9eb0-8aeae1899606/volumes"
Oct 31 02:08:02.976902 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:02.976857    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8644d884b7-fgzql" event={"ID":"19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec","Type":"ContainerStarted","Data":"359f194a68d6cd66882e30402c7b9224003b00199ebeb5e11b02590df841627c"}
Oct 31 02:08:02.977376 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:02.976950    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-8644d884b7-fgzql"
Oct 31 02:08:03.001022 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:03.000968    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-8644d884b7-fgzql" podStartSLOduration=6.000953684 podStartE2EDuration="6.000953684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:08:02.999193131 +0000 UTC m=+2839.002907422" watchObservedRunningTime="2026-10-31 02:08:03.000953684 +0000 UTC m=+2839.004667995"
Oct 31 02:08:04.719140 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:04.719104    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8644d884b7-fgzql"]
Oct 31 02:08:07.021576 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:07.021524    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:08:10.091483 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.091401    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-8644d884b7-fgzql" podUID="19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec" containerName="payload-processing" containerID="cri-o://359f194a68d6cd66882e30402c7b9224003b00199ebeb5e11b02590df841627c" gracePeriod=25
Oct 31 02:08:10.447260 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.447221    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8644d884b7-fgzql"
Oct 31 02:08:10.488289 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.488245    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-56d598494c-r5cmp"]
Oct 31 02:08:10.489503 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.489473    5650 cpu_manager.go:414] "containerMap: removing container" podUID="19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec" containerName="payload-processing"
Oct 31 02:08:10.489503 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.489501    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec" containerName="payload-processing"
Oct 31 02:08:10.489791 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.489763    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-provider-credentials\" (UniqueName: \"kubernetes.io/projected/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-provider-credentials\") pod \"19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec\" (UID: \"19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec\") "
Oct 31 02:08:10.489873 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.489806    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec" containerName="payload-processing"
Oct 31 02:08:10.489873 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.489826    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-plugins-config-volume\") pod \"19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec\" (UID: \"19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec\") "
Oct 31 02:08:10.489988 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.489941    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-routing-overlay-volume\") pod \"19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec\" (UID: \"19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec\") "
Oct 31 02:08:10.490780 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.490042    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-kube-api-access-fwpq6\" (UniqueName: \"kubernetes.io/projected/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-kube-api-access-fwpq6\") pod \"19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec\" (UID: \"19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec\") "
Oct 31 02:08:10.491981 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.491948    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-routing-overlay-volume" pod "19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec" (UID: "19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:08:10.492696 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.492293    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-plugins-config-volume" pod "19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec" (UID: "19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:08:10.493973 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.493946    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-kube-api-access-fwpq6" pod "19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec" (UID: "19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec"). InnerVolumeSpecName "kube-api-access-fwpq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:10.494245 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.494221    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-provider-credentials" pod "19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec" (UID: "19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:10.497016 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.496975    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56d598494c-r5cmp"
Oct 31 02:08:10.502201 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.502172    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fd4dcad8fadb\""
Oct 31 02:08:10.502347 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.502250    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f3e2ace89eb9\""
Oct 31 02:08:10.510187 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.510156    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56d598494c-r5cmp"]
Oct 31 02:08:10.592583 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.592548    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/371ad7a5-5afc-45a9-9dcb-728d73022f67-routing-overlay-volume\") pod \"payload-processing-external-model-56d598494c-r5cmp\" (UID: \"371ad7a5-5afc-45a9-9dcb-728d73022f67\") " pod="models-as-a-service/payload-processing-external-model-56d598494c-r5cmp"
Oct 31 02:08:10.592738 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.592602    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/371ad7a5-5afc-45a9-9dcb-728d73022f67-provider-credentials\") pod \"payload-processing-external-model-56d598494c-r5cmp\" (UID: \"371ad7a5-5afc-45a9-9dcb-728d73022f67\") " pod="models-as-a-service/payload-processing-external-model-56d598494c-r5cmp"
Oct 31 02:08:10.592738 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.592717    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371ad7a5-5afc-45a9-9dcb-728d73022f67-plugins-config-volume\") pod \"payload-processing-external-model-56d598494c-r5cmp\" (UID: \"371ad7a5-5afc-45a9-9dcb-728d73022f67\") " pod="models-as-a-service/payload-processing-external-model-56d598494c-r5cmp"
Oct 31 02:08:10.592853 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.592753    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8w8d\" (UniqueName: \"kubernetes.io/projected/371ad7a5-5afc-45a9-9dcb-728d73022f67-kube-api-access-q8w8d\") pod \"payload-processing-external-model-56d598494c-r5cmp\" (UID: \"371ad7a5-5afc-45a9-9dcb-728d73022f67\") " pod="models-as-a-service/payload-processing-external-model-56d598494c-r5cmp"
Oct 31 02:08:10.592951 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.592931    5650 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-provider-credentials\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:08:10.592994 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.592956    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:08:10.592994 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.592968    5650 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-routing-overlay-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:08:10.592994 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.592977    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwpq6\" (UniqueName: \"kubernetes.io/projected/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec-kube-api-access-fwpq6\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:08:10.695137 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.695046    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/371ad7a5-5afc-45a9-9dcb-728d73022f67-routing-overlay-volume\") pod \"payload-processing-external-model-56d598494c-r5cmp\" (UID: \"371ad7a5-5afc-45a9-9dcb-728d73022f67\") " pod="models-as-a-service/payload-processing-external-model-56d598494c-r5cmp"
Oct 31 02:08:10.695137 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.695098    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/371ad7a5-5afc-45a9-9dcb-728d73022f67-provider-credentials\") pod \"payload-processing-external-model-56d598494c-r5cmp\" (UID: \"371ad7a5-5afc-45a9-9dcb-728d73022f67\") " pod="models-as-a-service/payload-processing-external-model-56d598494c-r5cmp"
Oct 31 02:08:10.695355 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.695145    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371ad7a5-5afc-45a9-9dcb-728d73022f67-plugins-config-volume\") pod \"payload-processing-external-model-56d598494c-r5cmp\" (UID: \"371ad7a5-5afc-45a9-9dcb-728d73022f67\") " pod="models-as-a-service/payload-processing-external-model-56d598494c-r5cmp"
Oct 31 02:08:10.695355 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.695170    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8w8d\" (UniqueName: \"kubernetes.io/projected/371ad7a5-5afc-45a9-9dcb-728d73022f67-kube-api-access-q8w8d\") pod \"payload-processing-external-model-56d598494c-r5cmp\" (UID: \"371ad7a5-5afc-45a9-9dcb-728d73022f67\") " pod="models-as-a-service/payload-processing-external-model-56d598494c-r5cmp"
Oct 31 02:08:10.696310 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.696279    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/371ad7a5-5afc-45a9-9dcb-728d73022f67-routing-overlay-volume\") pod \"payload-processing-external-model-56d598494c-r5cmp\" (UID: \"371ad7a5-5afc-45a9-9dcb-728d73022f67\") " pod="models-as-a-service/payload-processing-external-model-56d598494c-r5cmp"
Oct 31 02:08:10.696999 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.696510    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371ad7a5-5afc-45a9-9dcb-728d73022f67-plugins-config-volume\") pod \"payload-processing-external-model-56d598494c-r5cmp\" (UID: \"371ad7a5-5afc-45a9-9dcb-728d73022f67\") " pod="models-as-a-service/payload-processing-external-model-56d598494c-r5cmp"
Oct 31 02:08:10.698544 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.698520    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/371ad7a5-5afc-45a9-9dcb-728d73022f67-provider-credentials\") pod \"payload-processing-external-model-56d598494c-r5cmp\" (UID: \"371ad7a5-5afc-45a9-9dcb-728d73022f67\") " pod="models-as-a-service/payload-processing-external-model-56d598494c-r5cmp"
Oct 31 02:08:10.708674 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.708652    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8w8d\" (UniqueName: \"kubernetes.io/projected/371ad7a5-5afc-45a9-9dcb-728d73022f67-kube-api-access-q8w8d\") pod \"payload-processing-external-model-56d598494c-r5cmp\" (UID: \"371ad7a5-5afc-45a9-9dcb-728d73022f67\") " pod="models-as-a-service/payload-processing-external-model-56d598494c-r5cmp"
Oct 31 02:08:10.734281 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.734216    5650 generic.go:356] "Generic (PLEG): container finished" podID="19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec" containerID="359f194a68d6cd66882e30402c7b9224003b00199ebeb5e11b02590df841627c" exitCode=0
Oct 31 02:08:10.734539 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.734516    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8644d884b7-fgzql" event={"ID":"19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec","Type":"ContainerDied","Data":"359f194a68d6cd66882e30402c7b9224003b00199ebeb5e11b02590df841627c"}
Oct 31 02:08:10.734600 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.734584    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8644d884b7-fgzql" event={"ID":"19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec","Type":"ContainerDied","Data":"012cc2df07ae4452266305997d5adcbff87d91571021dfbd51277e09b964ad56"}
Oct 31 02:08:10.734642 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.734616    5650 scope.go:122] "RemoveContainer" containerID="359f194a68d6cd66882e30402c7b9224003b00199ebeb5e11b02590df841627c"
Oct 31 02:08:10.734778 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.734761    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8644d884b7-fgzql"
Oct 31 02:08:10.748326 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.748302    5650 scope.go:122] "RemoveContainer" containerID="359f194a68d6cd66882e30402c7b9224003b00199ebeb5e11b02590df841627c"
Oct 31 02:08:10.748652 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:08:10.748631    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"359f194a68d6cd66882e30402c7b9224003b00199ebeb5e11b02590df841627c\": container with ID starting with 359f194a68d6cd66882e30402c7b9224003b00199ebeb5e11b02590df841627c not found: ID does not exist" containerID="359f194a68d6cd66882e30402c7b9224003b00199ebeb5e11b02590df841627c"
Oct 31 02:08:10.748709 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.748661    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359f194a68d6cd66882e30402c7b9224003b00199ebeb5e11b02590df841627c"} err="failed to get container status \"359f194a68d6cd66882e30402c7b9224003b00199ebeb5e11b02590df841627c\": rpc error: code = NotFound desc = could not find container \"359f194a68d6cd66882e30402c7b9224003b00199ebeb5e11b02590df841627c\": container with ID starting with 359f194a68d6cd66882e30402c7b9224003b00199ebeb5e11b02590df841627c not found: ID does not exist"
Oct 31 02:08:10.774046 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.774007    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8644d884b7-fgzql"]
Oct 31 02:08:10.779323 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.779294    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-8644d884b7-fgzql"]
Oct 31 02:08:10.865989 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:10.865930    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56d598494c-r5cmp"
Oct 31 02:08:11.175229 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:08:11.175199    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371ad7a5_5afc_45a9_9dcb_728d73022f67.slice/crio-8e624987fb81ada9960b7ed8e86aaf6df0c944145100fc754381672b4bce3d23 WatchSource:0}: Error finding container 8e624987fb81ada9960b7ed8e86aaf6df0c944145100fc754381672b4bce3d23: Status 404 returned error can't find the container with id 8e624987fb81ada9960b7ed8e86aaf6df0c944145100fc754381672b4bce3d23
Oct 31 02:08:11.175576 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:11.175554    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56d598494c-r5cmp"]
Oct 31 02:08:11.746613 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:11.746574    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56d598494c-r5cmp" event={"ID":"371ad7a5-5afc-45a9-9dcb-728d73022f67","Type":"ContainerStarted","Data":"375a7127ab9ea2a9a5b09dfe0a7c72b5784c14de39e8011f36e5f94b714b9a70"}
Oct 31 02:08:11.746834 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:11.746644    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56d598494c-r5cmp" event={"ID":"371ad7a5-5afc-45a9-9dcb-728d73022f67","Type":"ContainerStarted","Data":"8e624987fb81ada9960b7ed8e86aaf6df0c944145100fc754381672b4bce3d23"}
Oct 31 02:08:11.746834 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:11.746677    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-56d598494c-r5cmp"
Oct 31 02:08:11.770536 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:11.770486    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-56d598494c-r5cmp" podStartSLOduration=3.770431619 podStartE2EDuration="3.770431619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:08:11.768305419 +0000 UTC m=+2847.772019711" watchObservedRunningTime="2026-10-31 02:08:11.770431619 +0000 UTC m=+2847.774145910"
Oct 31 02:08:12.123215 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:12.123131    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec" path="/var/lib/kubelet/pods/19f93a29-bc3a-4f14-af5a-7f33a0d5b1ec/volumes"
Oct 31 02:08:13.555905 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:13.548373    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56d598494c-r5cmp"]
Oct 31 02:08:18.767897 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:18.767810    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-56d598494c-r5cmp" podUID="371ad7a5-5afc-45a9-9dcb-728d73022f67" containerName="payload-processing" containerID="cri-o://375a7127ab9ea2a9a5b09dfe0a7c72b5784c14de39e8011f36e5f94b714b9a70" gracePeriod=25
Oct 31 02:08:19.929828 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:19.929788    5650 generic.go:356] "Generic (PLEG): container finished" podID="371ad7a5-5afc-45a9-9dcb-728d73022f67" containerID="375a7127ab9ea2a9a5b09dfe0a7c72b5784c14de39e8011f36e5f94b714b9a70" exitCode=0
Oct 31 02:08:19.930300 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:19.929856    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56d598494c-r5cmp" event={"ID":"371ad7a5-5afc-45a9-9dcb-728d73022f67","Type":"ContainerDied","Data":"375a7127ab9ea2a9a5b09dfe0a7c72b5784c14de39e8011f36e5f94b714b9a70"}
Oct 31 02:08:19.930300 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:19.929915    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56d598494c-r5cmp" event={"ID":"371ad7a5-5afc-45a9-9dcb-728d73022f67","Type":"ContainerDied","Data":"8e624987fb81ada9960b7ed8e86aaf6df0c944145100fc754381672b4bce3d23"}
Oct 31 02:08:19.930300 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:19.929934    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e624987fb81ada9960b7ed8e86aaf6df0c944145100fc754381672b4bce3d23"
Oct 31 02:08:19.934979 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:19.934951    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56d598494c-r5cmp"
Oct 31 02:08:19.984584 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:19.984542    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh"]
Oct 31 02:08:19.987834 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:19.985996    5650 cpu_manager.go:414] "containerMap: removing container" podUID="371ad7a5-5afc-45a9-9dcb-728d73022f67" containerName="payload-processing"
Oct 31 02:08:19.987834 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:19.986022    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="371ad7a5-5afc-45a9-9dcb-728d73022f67" containerName="payload-processing"
Oct 31 02:08:19.987834 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:19.986235    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="371ad7a5-5afc-45a9-9dcb-728d73022f67" containerName="payload-processing"
Oct 31 02:08:20.006263 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:19.996855    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh"
Oct 31 02:08:20.006263 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.002780    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh"]
Oct 31 02:08:20.091679 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.090606    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/371ad7a5-5afc-45a9-9dcb-728d73022f67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371ad7a5-5afc-45a9-9dcb-728d73022f67-plugins-config-volume\") pod \"371ad7a5-5afc-45a9-9dcb-728d73022f67\" (UID: \"371ad7a5-5afc-45a9-9dcb-728d73022f67\") "
Oct 31 02:08:20.092038 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.091992    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/371ad7a5-5afc-45a9-9dcb-728d73022f67-kube-api-access-q8w8d\" (UniqueName: \"kubernetes.io/projected/371ad7a5-5afc-45a9-9dcb-728d73022f67-kube-api-access-q8w8d\") pod \"371ad7a5-5afc-45a9-9dcb-728d73022f67\" (UID: \"371ad7a5-5afc-45a9-9dcb-728d73022f67\") "
Oct 31 02:08:20.092373 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.092282    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/371ad7a5-5afc-45a9-9dcb-728d73022f67-provider-credentials\" (UniqueName: \"kubernetes.io/projected/371ad7a5-5afc-45a9-9dcb-728d73022f67-provider-credentials\") pod \"371ad7a5-5afc-45a9-9dcb-728d73022f67\" (UID: \"371ad7a5-5afc-45a9-9dcb-728d73022f67\") "
Oct 31 02:08:20.092566 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.092539    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/371ad7a5-5afc-45a9-9dcb-728d73022f67-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/371ad7a5-5afc-45a9-9dcb-728d73022f67-routing-overlay-volume\") pod \"371ad7a5-5afc-45a9-9dcb-728d73022f67\" (UID: \"371ad7a5-5afc-45a9-9dcb-728d73022f67\") "
Oct 31 02:08:20.107554 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.099629    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371ad7a5-5afc-45a9-9dcb-728d73022f67-plugins-config-volume" pod "371ad7a5-5afc-45a9-9dcb-728d73022f67" (UID: "371ad7a5-5afc-45a9-9dcb-728d73022f67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:08:20.107554 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.101420    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371ad7a5-5afc-45a9-9dcb-728d73022f67-routing-overlay-volume" pod "371ad7a5-5afc-45a9-9dcb-728d73022f67" (UID: "371ad7a5-5afc-45a9-9dcb-728d73022f67"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:08:20.107554 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.104721    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-plugins-config-volume\") pod \"payload-processing-external-model-d59bf7996-xb4nh\" (UID: \"4715d2d2-3d96-42e3-bc9c-ee9018c6d03e\") " pod="models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh"
Oct 31 02:08:20.107554 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.104937    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-provider-credentials\") pod \"payload-processing-external-model-d59bf7996-xb4nh\" (UID: \"4715d2d2-3d96-42e3-bc9c-ee9018c6d03e\") " pod="models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh"
Oct 31 02:08:20.107554 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.105239    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4zkw\" (UniqueName: \"kubernetes.io/projected/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-kube-api-access-z4zkw\") pod \"payload-processing-external-model-d59bf7996-xb4nh\" (UID: \"4715d2d2-3d96-42e3-bc9c-ee9018c6d03e\") " pod="models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh"
Oct 31 02:08:20.107554 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.105327    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-routing-overlay-volume\") pod \"payload-processing-external-model-d59bf7996-xb4nh\" (UID: \"4715d2d2-3d96-42e3-bc9c-ee9018c6d03e\") " pod="models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh"
Oct 31 02:08:20.107554 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.105583    5650 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/371ad7a5-5afc-45a9-9dcb-728d73022f67-routing-overlay-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:08:20.107554 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.105620    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371ad7a5-5afc-45a9-9dcb-728d73022f67-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:08:20.116093 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.116055    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371ad7a5-5afc-45a9-9dcb-728d73022f67-provider-credentials" pod "371ad7a5-5afc-45a9-9dcb-728d73022f67" (UID: "371ad7a5-5afc-45a9-9dcb-728d73022f67"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:20.130292 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.128928    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371ad7a5-5afc-45a9-9dcb-728d73022f67-kube-api-access-q8w8d" pod "371ad7a5-5afc-45a9-9dcb-728d73022f67" (UID: "371ad7a5-5afc-45a9-9dcb-728d73022f67"). InnerVolumeSpecName "kube-api-access-q8w8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:20.207137 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.207097    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-provider-credentials\") pod \"payload-processing-external-model-d59bf7996-xb4nh\" (UID: \"4715d2d2-3d96-42e3-bc9c-ee9018c6d03e\") " pod="models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh"
Oct 31 02:08:20.207352 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.207300    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4zkw\" (UniqueName: \"kubernetes.io/projected/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-kube-api-access-z4zkw\") pod \"payload-processing-external-model-d59bf7996-xb4nh\" (UID: \"4715d2d2-3d96-42e3-bc9c-ee9018c6d03e\") " pod="models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh"
Oct 31 02:08:20.208093 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.207355    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-routing-overlay-volume\") pod \"payload-processing-external-model-d59bf7996-xb4nh\" (UID: \"4715d2d2-3d96-42e3-bc9c-ee9018c6d03e\") " pod="models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh"
Oct 31 02:08:20.208093 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.207576    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-plugins-config-volume\") pod \"payload-processing-external-model-d59bf7996-xb4nh\" (UID: \"4715d2d2-3d96-42e3-bc9c-ee9018c6d03e\") " pod="models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh"
Oct 31 02:08:20.208093 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.207750    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8w8d\" (UniqueName: \"kubernetes.io/projected/371ad7a5-5afc-45a9-9dcb-728d73022f67-kube-api-access-q8w8d\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:08:20.208093 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.207770    5650 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/371ad7a5-5afc-45a9-9dcb-728d73022f67-provider-credentials\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:08:20.208742 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.208310    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-routing-overlay-volume\") pod \"payload-processing-external-model-d59bf7996-xb4nh\" (UID: \"4715d2d2-3d96-42e3-bc9c-ee9018c6d03e\") " pod="models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh"
Oct 31 02:08:20.208990 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.208964    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-plugins-config-volume\") pod \"payload-processing-external-model-d59bf7996-xb4nh\" (UID: \"4715d2d2-3d96-42e3-bc9c-ee9018c6d03e\") " pod="models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh"
Oct 31 02:08:20.210301 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.210274    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-provider-credentials\") pod \"payload-processing-external-model-d59bf7996-xb4nh\" (UID: \"4715d2d2-3d96-42e3-bc9c-ee9018c6d03e\") " pod="models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh"
Oct 31 02:08:20.223332 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.223295    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4zkw\" (UniqueName: \"kubernetes.io/projected/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-kube-api-access-z4zkw\") pod \"payload-processing-external-model-d59bf7996-xb4nh\" (UID: \"4715d2d2-3d96-42e3-bc9c-ee9018c6d03e\") " pod="models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh"
Oct 31 02:08:20.372839 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.372755    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh"
Oct 31 02:08:20.733544 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:08:20.733511    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4715d2d2_3d96_42e3_bc9c_ee9018c6d03e.slice/crio-e325b1efe3ce289b1267a54ab9196e9c1d13f0dad41f9f4f40499d3beb366193 WatchSource:0}: Error finding container e325b1efe3ce289b1267a54ab9196e9c1d13f0dad41f9f4f40499d3beb366193: Status 404 returned error can't find the container with id e325b1efe3ce289b1267a54ab9196e9c1d13f0dad41f9f4f40499d3beb366193
Oct 31 02:08:20.734979 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.734954    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh"]
Oct 31 02:08:20.941015 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.940981    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh" event={"ID":"4715d2d2-3d96-42e3-bc9c-ee9018c6d03e","Type":"ContainerStarted","Data":"59a1c63f146f80318c4602868affe23ae7a82d6f5924a8d9cb54f0cc05084d9f"}
Oct 31 02:08:20.942237 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.941052    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh" event={"ID":"4715d2d2-3d96-42e3-bc9c-ee9018c6d03e","Type":"ContainerStarted","Data":"e325b1efe3ce289b1267a54ab9196e9c1d13f0dad41f9f4f40499d3beb366193"}
Oct 31 02:08:20.942237 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.941083    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh"
Oct 31 02:08:20.942237 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.941229    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56d598494c-r5cmp"
Oct 31 02:08:20.967786 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.967734    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh" podStartSLOduration=7.96771889 podStartE2EDuration="7.96771889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:08:20.96299099 +0000 UTC m=+2856.966705285" watchObservedRunningTime="2026-10-31 02:08:20.96771889 +0000 UTC m=+2856.971433259"
Oct 31 02:08:20.984752 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.984669    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56d598494c-r5cmp"]
Oct 31 02:08:20.990968 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:20.990941    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-56d598494c-r5cmp"]
Oct 31 02:08:22.123036 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:22.122994    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="371ad7a5-5afc-45a9-9dcb-728d73022f67" path="/var/lib/kubelet/pods/371ad7a5-5afc-45a9-9dcb-728d73022f67/volumes"
Oct 31 02:08:23.795329 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:23.793872    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh"]
Oct 31 02:08:28.570969 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:08:28.570925    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:08:28.794906 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:28.794863    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh" podUID="4715d2d2-3d96-42e3-bc9c-ee9018c6d03e" containerName="payload-processing" containerID="cri-o://59a1c63f146f80318c4602868affe23ae7a82d6f5924a8d9cb54f0cc05084d9f" gracePeriod=25
Oct 31 02:08:29.482298 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.482263    5650 generic.go:356] "Generic (PLEG): container finished" podID="4715d2d2-3d96-42e3-bc9c-ee9018c6d03e" containerID="59a1c63f146f80318c4602868affe23ae7a82d6f5924a8d9cb54f0cc05084d9f" exitCode=0
Oct 31 02:08:29.483064 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.483038    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh" event={"ID":"4715d2d2-3d96-42e3-bc9c-ee9018c6d03e","Type":"ContainerDied","Data":"59a1c63f146f80318c4602868affe23ae7a82d6f5924a8d9cb54f0cc05084d9f"}
Oct 31 02:08:29.484008 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.483984    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh"
Oct 31 02:08:29.694472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.671117    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-routing-overlay-volume\") pod \"4715d2d2-3d96-42e3-bc9c-ee9018c6d03e\" (UID: \"4715d2d2-3d96-42e3-bc9c-ee9018c6d03e\") "
Oct 31 02:08:29.716615 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.704842    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-kube-api-access-z4zkw\" (UniqueName: \"kubernetes.io/projected/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-kube-api-access-z4zkw\") pod \"4715d2d2-3d96-42e3-bc9c-ee9018c6d03e\" (UID: \"4715d2d2-3d96-42e3-bc9c-ee9018c6d03e\") "
Oct 31 02:08:29.717313 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.717280    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-plugins-config-volume\") pod \"4715d2d2-3d96-42e3-bc9c-ee9018c6d03e\" (UID: \"4715d2d2-3d96-42e3-bc9c-ee9018c6d03e\") "
Oct 31 02:08:29.732768 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.732670    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-provider-credentials\") pod \"4715d2d2-3d96-42e3-bc9c-ee9018c6d03e\" (UID: \"4715d2d2-3d96-42e3-bc9c-ee9018c6d03e\") "
Oct 31 02:08:29.749533 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.675105    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-routing-overlay-volume" pod "4715d2d2-3d96-42e3-bc9c-ee9018c6d03e" (UID: "4715d2d2-3d96-42e3-bc9c-ee9018c6d03e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:08:29.822552 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.717947    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-plugins-config-volume" pod "4715d2d2-3d96-42e3-bc9c-ee9018c6d03e" (UID: "4715d2d2-3d96-42e3-bc9c-ee9018c6d03e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:08:29.822865 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.805987    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw"]
Oct 31 02:08:29.871470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.864633    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:08:29.871470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.864669    5650 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-routing-overlay-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:08:29.926706 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.926666    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-kube-api-access-z4zkw" pod "4715d2d2-3d96-42e3-bc9c-ee9018c6d03e" (UID: "4715d2d2-3d96-42e3-bc9c-ee9018c6d03e"). InnerVolumeSpecName "kube-api-access-z4zkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:29.937384 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.937347    5650 cpu_manager.go:414] "containerMap: removing container" podUID="4715d2d2-3d96-42e3-bc9c-ee9018c6d03e" containerName="payload-processing"
Oct 31 02:08:29.939366 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.939338    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4715d2d2-3d96-42e3-bc9c-ee9018c6d03e" containerName="payload-processing"
Oct 31 02:08:29.940916 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.940890    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="4715d2d2-3d96-42e3-bc9c-ee9018c6d03e" containerName="payload-processing"
Oct 31 02:08:29.950395 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.950371    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw"
Oct 31 02:08:29.963147 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.960838    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ac7e0d\"/\"kube-root-ca.crt\""
Oct 31 02:08:29.967614 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.963301    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw"]
Oct 31 02:08:29.967614 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.965365    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ac7e0d\"/\"routing-overlay\""
Oct 31 02:08:29.967614 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.965473    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ac7e0d\"/\"agc-cred-3b375805fa8e\""
Oct 31 02:08:29.967614 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.965541    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ac7e0d\"/\"openshift-service-ca.crt\""
Oct 31 02:08:29.967614 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.965688    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ac7e0d\"/\"payload-processing-external-model-e2e--1ed73e85-dockercfg-mbxtd\""
Oct 31 02:08:29.967614 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.965770    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ac7e0d\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ac7e0d\""
Oct 31 02:08:29.985168 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.967845    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4zkw\" (UniqueName: \"kubernetes.io/projected/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-kube-api-access-z4zkw\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:08:29.998985 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:29.997305    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-provider-credentials" pod "4715d2d2-3d96-42e3-bc9c-ee9018c6d03e" (UID: "4715d2d2-3d96-42e3-bc9c-ee9018c6d03e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:30.085538 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:30.072468    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw\" (UID: \"6e3c5fc2-200f-4c69-bfcd-c223556a4acb\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw"
Oct 31 02:08:30.086115 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:30.086079    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw\" (UID: \"6e3c5fc2-200f-4c69-bfcd-c223556a4acb\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw"
Oct 31 02:08:30.086737 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:30.086709    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw\" (UID: \"6e3c5fc2-200f-4c69-bfcd-c223556a4acb\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw"
Oct 31 02:08:30.087023 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:30.086999    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2dmr\" (UniqueName: \"kubernetes.io/projected/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-kube-api-access-h2dmr\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw\" (UID: \"6e3c5fc2-200f-4c69-bfcd-c223556a4acb\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw"
Oct 31 02:08:30.087351 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:30.087330    5650 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e-provider-credentials\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:08:30.189621 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:30.189577    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw\" (UID: \"6e3c5fc2-200f-4c69-bfcd-c223556a4acb\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw"
Oct 31 02:08:30.189832 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:30.189652    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2dmr\" (UniqueName: \"kubernetes.io/projected/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-kube-api-access-h2dmr\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw\" (UID: \"6e3c5fc2-200f-4c69-bfcd-c223556a4acb\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw"
Oct 31 02:08:30.189832 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:30.189734    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw\" (UID: \"6e3c5fc2-200f-4c69-bfcd-c223556a4acb\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw"
Oct 31 02:08:30.189982 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:30.189909    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw\" (UID: \"6e3c5fc2-200f-4c69-bfcd-c223556a4acb\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw"
Oct 31 02:08:30.190628 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:30.190595    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw\" (UID: \"6e3c5fc2-200f-4c69-bfcd-c223556a4acb\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw"
Oct 31 02:08:30.190932 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:30.190910    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw\" (UID: \"6e3c5fc2-200f-4c69-bfcd-c223556a4acb\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw"
Oct 31 02:08:30.193787 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:30.193759    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw\" (UID: \"6e3c5fc2-200f-4c69-bfcd-c223556a4acb\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw"
Oct 31 02:08:30.207857 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:30.207820    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2dmr\" (UniqueName: \"kubernetes.io/projected/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-kube-api-access-h2dmr\") pod \"payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw\" (UID: \"6e3c5fc2-200f-4c69-bfcd-c223556a4acb\") " pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw"
Oct 31 02:08:30.346020 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:30.345694    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw"
Oct 31 02:08:30.519480 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:30.515846    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh" event={"ID":"4715d2d2-3d96-42e3-bc9c-ee9018c6d03e","Type":"ContainerDied","Data":"e325b1efe3ce289b1267a54ab9196e9c1d13f0dad41f9f4f40499d3beb366193"}
Oct 31 02:08:30.519480 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:30.515922    5650 scope.go:122] "RemoveContainer" containerID="59a1c63f146f80318c4602868affe23ae7a82d6f5924a8d9cb54f0cc05084d9f"
Oct 31 02:08:30.519480 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:30.516108    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh"
Oct 31 02:08:30.670518 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:30.667604    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh"]
Oct 31 02:08:30.769466 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:30.748791    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-d59bf7996-xb4nh"]
Oct 31 02:08:31.379509 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:08:31.379391    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e3c5fc2_200f_4c69_bfcd_c223556a4acb.slice/crio-45d65105d781803d35c8cc65abdb4c0c55a569e5d32d74c5b37ce30b2ff80aab WatchSource:0}: Error finding container 45d65105d781803d35c8cc65abdb4c0c55a569e5d32d74c5b37ce30b2ff80aab: Status 404 returned error can't find the container with id 45d65105d781803d35c8cc65abdb4c0c55a569e5d32d74c5b37ce30b2ff80aab
Oct 31 02:08:31.381562 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:31.381529    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw"]
Oct 31 02:08:31.532415 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:31.532368    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw" event={"ID":"6e3c5fc2-200f-4c69-bfcd-c223556a4acb","Type":"ContainerStarted","Data":"6d36ab70d0a59f6241c233c6c7c0916a10a61c5efcc481066b4c134b495036cb"}
Oct 31 02:08:31.532680 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:31.532514    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw" event={"ID":"6e3c5fc2-200f-4c69-bfcd-c223556a4acb","Type":"ContainerStarted","Data":"45d65105d781803d35c8cc65abdb4c0c55a569e5d32d74c5b37ce30b2ff80aab"}
Oct 31 02:08:31.533582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:31.533555    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw"
Oct 31 02:08:31.561480 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:31.561325    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw" podStartSLOduration=13.561309854 podStartE2EDuration="13.561309854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:08:31.560878978 +0000 UTC m=+2867.564593269" watchObservedRunningTime="2026-10-31 02:08:31.561309854 +0000 UTC m=+2867.565024145"
Oct 31 02:08:32.127577 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:32.127418    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4715d2d2-3d96-42e3-bc9c-ee9018c6d03e" path="/var/lib/kubelet/pods/4715d2d2-3d96-42e3-bc9c-ee9018c6d03e/volumes"
Oct 31 02:08:42.613013 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:08:42.612964    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw"
Oct 31 02:08:58.563728 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:08:58.563642    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:09:28.564190 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:09:28.564148    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:09:38.035890 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:09:38.035850    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01be29b0-qll8s"]
Oct 31 02:09:38.040968 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:09:38.040942    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01be29b0-qll8s"]
Oct 31 02:09:38.143235 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:09:38.140092    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cef28068-9242-4bdc-95dd-ea7620723963" path="/var/lib/kubelet/pods/cef28068-9242-4bdc-95dd-ea7620723963/volumes"
Oct 31 02:09:40.027548 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:09:40.027497    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f45dfc5-8pnkk"]
Oct 31 02:09:40.033345 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:09:40.033309    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f45dfc5-8pnkk"]
Oct 31 02:09:40.124010 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:09:40.123976    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f47ea99-7261-436b-a2d1-7ab1d54da8a1" path="/var/lib/kubelet/pods/3f47ea99-7261-436b-a2d1-7ab1d54da8a1/volumes"
Oct 31 02:09:47.708609 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:09:47.708569    5650 scope.go:122] "RemoveContainer" containerID="bce58502d4104cc27917ddd412761daa62f816f7ef75f6b597f1a9037cb2e15a"
Oct 31 02:09:48.027265 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:09:48.027177    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da94df5d-gjgh6"]
Oct 31 02:09:48.032058 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:09:48.032033    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da94df5d-gjgh6"]
Oct 31 02:09:48.121853 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:09:48.121816    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18cc0eff-9057-45ef-8ee1-461d29daf8f9" path="/var/lib/kubelet/pods/18cc0eff-9057-45ef-8ee1-461d29daf8f9/volumes"
Oct 31 02:09:58.557514 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:09:58.557475    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:10:28.555275 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:10:28.555192    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:10:47.890675 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:10:47.890633    5650 scope.go:122] "RemoveContainer" containerID="00641a5ffe18415594361184c220ce59d37aa421613aba5e13292f6ef2309414"
Oct 31 02:10:47.904728 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:10:47.904700    5650 scope.go:122] "RemoveContainer" containerID="c66b6d9d57902a1de17f375c7e721db5e791dec06a8c50f5aa7bd04eee73f099"
Oct 31 02:10:47.917170 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:10:47.917146    5650 scope.go:122] "RemoveContainer" containerID="c9ae767bd1ed030cf3ceb458ed2102c9520b3f71bcd4b47083f7b96411646424"
Oct 31 02:10:58.560146 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:10:58.560068    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:11:28.555323 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:11:28.555276    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:11:58.739263 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:11:58.739174    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:12:17.048609 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:17.046426    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw"]
Oct 31 02:12:19.827935 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:19.827823    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"]
Oct 31 02:12:19.828546 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:19.828140    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg" podUID="cd4d7677-6d53-42e9-8416-024a78dc3ff7" containerName="istio-proxy" containerID="cri-o://6e5dbee1ee1c45eeec16a37cbd6b37e6d23d34d2028ec5d2dead95bf961e9031" gracePeriod=30
Oct 31 02:12:19.846822 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:19.846774    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"]
Oct 31 02:12:19.847798 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:19.847748    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98" podUID="b255f882-402f-4db2-900b-051813a4300a" containerName="istio-proxy" containerID="cri-o://6c36837539e4734e79cd5d6a380cf496e212645a8475ef34a23acdb876da0058" gracePeriod=30
Oct 31 02:12:22.050471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:22.049865    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw" podUID="6e3c5fc2-200f-4c69-bfcd-c223556a4acb" containerName="payload-processing" containerID="cri-o://6d36ab70d0a59f6241c233c6c7c0916a10a61c5efcc481066b4c134b495036cb" gracePeriod=25
Oct 31 02:12:22.288683 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:22.288648    5650 generic.go:356] "Generic (PLEG): container finished" podID="6e3c5fc2-200f-4c69-bfcd-c223556a4acb" containerID="6d36ab70d0a59f6241c233c6c7c0916a10a61c5efcc481066b4c134b495036cb" exitCode=0
Oct 31 02:12:22.288868 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:22.288712    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw" event={"ID":"6e3c5fc2-200f-4c69-bfcd-c223556a4acb","Type":"ContainerDied","Data":"6d36ab70d0a59f6241c233c6c7c0916a10a61c5efcc481066b4c134b495036cb"}
Oct 31 02:12:22.999065 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:22.999033    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw"
Oct 31 02:12:23.155518 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:23.155094    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-plugins-config-volume\") pod \"6e3c5fc2-200f-4c69-bfcd-c223556a4acb\" (UID: \"6e3c5fc2-200f-4c69-bfcd-c223556a4acb\") "
Oct 31 02:12:23.155518 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:23.155303    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-kube-api-access-h2dmr\" (UniqueName: \"kubernetes.io/projected/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-kube-api-access-h2dmr\") pod \"6e3c5fc2-200f-4c69-bfcd-c223556a4acb\" (UID: \"6e3c5fc2-200f-4c69-bfcd-c223556a4acb\") "
Oct 31 02:12:23.155518 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:23.155371    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-provider-credentials\") pod \"6e3c5fc2-200f-4c69-bfcd-c223556a4acb\" (UID: \"6e3c5fc2-200f-4c69-bfcd-c223556a4acb\") "
Oct 31 02:12:23.156167 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:23.155653    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-routing-overlay-volume\") pod \"6e3c5fc2-200f-4c69-bfcd-c223556a4acb\" (UID: \"6e3c5fc2-200f-4c69-bfcd-c223556a4acb\") "
Oct 31 02:12:23.156167 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:23.156051    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-plugins-config-volume" pod "6e3c5fc2-200f-4c69-bfcd-c223556a4acb" (UID: "6e3c5fc2-200f-4c69-bfcd-c223556a4acb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:12:23.158186 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:23.156670    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:23.158186 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:23.157908    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-routing-overlay-volume" pod "6e3c5fc2-200f-4c69-bfcd-c223556a4acb" (UID: "6e3c5fc2-200f-4c69-bfcd-c223556a4acb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:12:23.172513 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:23.165660    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-kube-api-access-h2dmr" pod "6e3c5fc2-200f-4c69-bfcd-c223556a4acb" (UID: "6e3c5fc2-200f-4c69-bfcd-c223556a4acb"). InnerVolumeSpecName "kube-api-access-h2dmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:12:23.172513 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:23.165819    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-provider-credentials" pod "6e3c5fc2-200f-4c69-bfcd-c223556a4acb" (UID: "6e3c5fc2-200f-4c69-bfcd-c223556a4acb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:12:23.316359 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:23.316324    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2dmr\" (UniqueName: \"kubernetes.io/projected/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-kube-api-access-h2dmr\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:23.316359 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:23.316357    5650 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-provider-credentials\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:23.316713 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:23.316371    5650 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e3c5fc2-200f-4c69-bfcd-c223556a4acb-routing-overlay-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:23.319775 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:23.319738    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw" event={"ID":"6e3c5fc2-200f-4c69-bfcd-c223556a4acb","Type":"ContainerDied","Data":"45d65105d781803d35c8cc65abdb4c0c55a569e5d32d74c5b37ce30b2ff80aab"}
Oct 31 02:12:23.319942 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:23.319812    5650 scope.go:122] "RemoveContainer" containerID="6d36ab70d0a59f6241c233c6c7c0916a10a61c5efcc481066b4c134b495036cb"
Oct 31 02:12:23.320014 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:23.319975    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw"
Oct 31 02:12:23.356942 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:23.356906    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw"]
Oct 31 02:12:23.363747 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:23.363716    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw"]
Oct 31 02:12:23.627795 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:23.627745    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-ac7e0d/payload-processing-external-model-e2e-extmod-a-ac7e0d-7688cjlhw" podUID="6e3c5fc2-200f-4c69-bfcd-c223556a4acb" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.240:50052\" within 1s: context deadline exceeded"
Oct 31 02:12:23.627988 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:12:23.627835    5650 logging.go:55] [core] [Channel #8476 SubChannel #8477]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.240:50052", ServerName: "10.134.0.240:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.240:50052: operation was canceled"
Oct 31 02:12:24.124962 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:24.124920    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e3c5fc2-200f-4c69-bfcd-c223556a4acb" path="/var/lib/kubelet/pods/6e3c5fc2-200f-4c69-bfcd-c223556a4acb/volumes"
Oct 31 02:12:25.239511 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.239477    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:12:25.273555 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.273524    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-workload-certs\") pod \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") "
Oct 31 02:12:25.273736 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.273703    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istiod-ca-cert\") pod \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") "
Oct 31 02:12:25.273839 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.273765    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-data\") pod \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") "
Oct 31 02:12:25.273839 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.273794    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd4d7677-6d53-42e9-8416-024a78dc3ff7-kube-api-access-j59rj\" (UniqueName: \"kubernetes.io/projected/cd4d7677-6d53-42e9-8416-024a78dc3ff7-kube-api-access-j59rj\") pod \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") "
Oct 31 02:12:25.281631 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.273956    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-envoy\") pod \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") "
Oct 31 02:12:25.281631 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.274064    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-workload-socket\") pod \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") "
Oct 31 02:12:25.281631 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.274196    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-podinfo\") pod \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") "
Oct 31 02:12:25.281631 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.274251    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-credential-socket\") pod \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") "
Oct 31 02:12:25.281631 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.274350    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-token\" (UniqueName: \"kubernetes.io/projected/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-token\") pod \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\" (UID: \"cd4d7677-6d53-42e9-8416-024a78dc3ff7\") "
Oct 31 02:12:25.281631 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.278007    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-data" pod "cd4d7677-6d53-42e9-8416-024a78dc3ff7" (UID: "cd4d7677-6d53-42e9-8416-024a78dc3ff7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:12:25.281631 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.278249    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-workload-certs" pod "cd4d7677-6d53-42e9-8416-024a78dc3ff7" (UID: "cd4d7677-6d53-42e9-8416-024a78dc3ff7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:12:25.281631 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.278336    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-credential-socket" pod "cd4d7677-6d53-42e9-8416-024a78dc3ff7" (UID: "cd4d7677-6d53-42e9-8416-024a78dc3ff7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:12:25.281631 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.279883    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istiod-ca-cert" pod "cd4d7677-6d53-42e9-8416-024a78dc3ff7" (UID: "cd4d7677-6d53-42e9-8416-024a78dc3ff7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:12:25.284159 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.281797    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd4d7677-6d53-42e9-8416-024a78dc3ff7-kube-api-access-j59rj" pod "cd4d7677-6d53-42e9-8416-024a78dc3ff7" (UID: "cd4d7677-6d53-42e9-8416-024a78dc3ff7"). InnerVolumeSpecName "kube-api-access-j59rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:12:25.286724 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.286699    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"]
Oct 31 02:12:25.287631 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.287603    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-envoy" pod "cd4d7677-6d53-42e9-8416-024a78dc3ff7" (UID: "cd4d7677-6d53-42e9-8416-024a78dc3ff7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:12:25.287722 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.287677    5650 cpu_manager.go:414] "containerMap: removing container" podUID="cd4d7677-6d53-42e9-8416-024a78dc3ff7" containerName="istio-proxy"
Oct 31 02:12:25.287722 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.287697    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd4d7677-6d53-42e9-8416-024a78dc3ff7" containerName="istio-proxy"
Oct 31 02:12:25.287722 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.287718    5650 cpu_manager.go:414] "containerMap: removing container" podUID="cd4d7677-6d53-42e9-8416-024a78dc3ff7" containerName="istio-proxy"
Oct 31 02:12:25.287844 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.287727    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd4d7677-6d53-42e9-8416-024a78dc3ff7" containerName="istio-proxy"
Oct 31 02:12:25.287844 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.287807    5650 cpu_manager.go:414] "containerMap: removing container" podUID="6e3c5fc2-200f-4c69-bfcd-c223556a4acb" containerName="payload-processing"
Oct 31 02:12:25.287844 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.287817    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e3c5fc2-200f-4c69-bfcd-c223556a4acb" containerName="payload-processing"
Oct 31 02:12:25.288021 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.288005    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e3c5fc2-200f-4c69-bfcd-c223556a4acb" containerName="payload-processing"
Oct 31 02:12:25.288208 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.288187    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd4d7677-6d53-42e9-8416-024a78dc3ff7" containerName="istio-proxy"
Oct 31 02:12:25.288208 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.288043    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-podinfo" pod "cd4d7677-6d53-42e9-8416-024a78dc3ff7" (UID: "cd4d7677-6d53-42e9-8416-024a78dc3ff7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:12:25.288338 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.288222    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-workload-socket" pod "cd4d7677-6d53-42e9-8416-024a78dc3ff7" (UID: "cd4d7677-6d53-42e9-8416-024a78dc3ff7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:12:25.288338 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.288208    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd4d7677-6d53-42e9-8416-024a78dc3ff7" containerName="istio-proxy"
Oct 31 02:12:25.288338 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.288076    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-token" pod "cd4d7677-6d53-42e9-8416-024a78dc3ff7" (UID: "cd4d7677-6d53-42e9-8416-024a78dc3ff7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:12:25.294341 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.294301    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.305174 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.303667    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:12:25.307350 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.306235    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"]
Oct 31 02:12:25.367458 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.367403    5650 generic.go:356] "Generic (PLEG): container finished" podID="cd4d7677-6d53-42e9-8416-024a78dc3ff7" containerID="6e5dbee1ee1c45eeec16a37cbd6b37e6d23d34d2028ec5d2dead95bf961e9031" exitCode=0
Oct 31 02:12:25.367785 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.367763    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"
Oct 31 02:12:25.369262 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.368055    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg" event={"ID":"cd4d7677-6d53-42e9-8416-024a78dc3ff7","Type":"ContainerDied","Data":"6e5dbee1ee1c45eeec16a37cbd6b37e6d23d34d2028ec5d2dead95bf961e9031"}
Oct 31 02:12:25.369262 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.368172    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg" event={"ID":"cd4d7677-6d53-42e9-8416-024a78dc3ff7","Type":"ContainerDied","Data":"6d5e18e2cdabd93bc5b89399d6f22a413ada0a16b733a9fc2ca412ebade18ccb"}
Oct 31 02:12:25.369262 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.368216    5650 scope.go:122] "RemoveContainer" containerID="6e5dbee1ee1c45eeec16a37cbd6b37e6d23d34d2028ec5d2dead95bf961e9031"
Oct 31 02:12:25.373506 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.373475    5650 generic.go:356] "Generic (PLEG): container finished" podID="b255f882-402f-4db2-900b-051813a4300a" containerID="6c36837539e4734e79cd5d6a380cf496e212645a8475ef34a23acdb876da0058" exitCode=0
Oct 31 02:12:25.383049 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.373792    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"
Oct 31 02:12:25.383477 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.374039    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98" event={"ID":"b255f882-402f-4db2-900b-051813a4300a","Type":"ContainerDied","Data":"6c36837539e4734e79cd5d6a380cf496e212645a8475ef34a23acdb876da0058"}
Oct 31 02:12:25.383709 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.383688    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98" event={"ID":"b255f882-402f-4db2-900b-051813a4300a","Type":"ContainerDied","Data":"5de09eab1b3dd91ad5f86a6995951e1bf5fb680e000c99d232ccea31f207ed81"}
Oct 31 02:12:25.396539 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.395954    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-istio-envoy\") pod \"b255f882-402f-4db2-900b-051813a4300a\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") "
Oct 31 02:12:25.396539 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.396118    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-workload-socket\") pod \"b255f882-402f-4db2-900b-051813a4300a\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") "
Oct 31 02:12:25.397523 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.397431    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-workload-socket" pod "b255f882-402f-4db2-900b-051813a4300a" (UID: "b255f882-402f-4db2-900b-051813a4300a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:12:25.398936 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.398907    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b255f882-402f-4db2-900b-051813a4300a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b255f882-402f-4db2-900b-051813a4300a-istiod-ca-cert\") pod \"b255f882-402f-4db2-900b-051813a4300a\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") "
Oct 31 02:12:25.399082 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.398969    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b255f882-402f-4db2-900b-051813a4300a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b255f882-402f-4db2-900b-051813a4300a-istio-podinfo\") pod \"b255f882-402f-4db2-900b-051813a4300a\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") "
Oct 31 02:12:25.399155 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.399100    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b255f882-402f-4db2-900b-051813a4300a-kube-api-access-lsl7j\" (UniqueName: \"kubernetes.io/projected/b255f882-402f-4db2-900b-051813a4300a-kube-api-access-lsl7j\") pod \"b255f882-402f-4db2-900b-051813a4300a\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") "
Oct 31 02:12:25.399155 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.399134    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b255f882-402f-4db2-900b-051813a4300a-istio-token\" (UniqueName: \"kubernetes.io/projected/b255f882-402f-4db2-900b-051813a4300a-istio-token\") pod \"b255f882-402f-4db2-900b-051813a4300a\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") "
Oct 31 02:12:25.399282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.399184    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-istio-data\") pod \"b255f882-402f-4db2-900b-051813a4300a\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") "
Oct 31 02:12:25.399282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.399275    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-credential-socket\") pod \"b255f882-402f-4db2-900b-051813a4300a\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") "
Oct 31 02:12:25.399406 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.399328    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-workload-certs\") pod \"b255f882-402f-4db2-900b-051813a4300a\" (UID: \"b255f882-402f-4db2-900b-051813a4300a\") "
Oct 31 02:12:25.401068 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.401038    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.401289 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.401268    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.401428 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.401411    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.402187 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.401572    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.402187 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.401711    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn98f\" (UniqueName: \"kubernetes.io/projected/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-kube-api-access-jn98f\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.402187 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.401776    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.402187 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.402023    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.402187 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.402046    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.402891 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.402388    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.402891 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.402633    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.402891 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.402653    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j59rj\" (UniqueName: \"kubernetes.io/projected/cd4d7677-6d53-42e9-8416-024a78dc3ff7-kube-api-access-j59rj\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.402891 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.402668    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.402891 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.402690    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.402891 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.402703    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.402891 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.402717    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.402891 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.402737    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.402891 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.402752    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd4d7677-6d53-42e9-8416-024a78dc3ff7-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.402891 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.402768    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.402891 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.402782    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd4d7677-6d53-42e9-8416-024a78dc3ff7-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.414554 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.414502    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b255f882-402f-4db2-900b-051813a4300a-istio-podinfo" pod "b255f882-402f-4db2-900b-051813a4300a" (UID: "b255f882-402f-4db2-900b-051813a4300a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:12:25.414965 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.414689    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-credential-socket" pod "b255f882-402f-4db2-900b-051813a4300a" (UID: "b255f882-402f-4db2-900b-051813a4300a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:12:25.414965 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.414783    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-istio-data" pod "b255f882-402f-4db2-900b-051813a4300a" (UID: "b255f882-402f-4db2-900b-051813a4300a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:12:25.414965 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.414841    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b255f882-402f-4db2-900b-051813a4300a-istio-token" pod "b255f882-402f-4db2-900b-051813a4300a" (UID: "b255f882-402f-4db2-900b-051813a4300a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:12:25.433352 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.415306    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b255f882-402f-4db2-900b-051813a4300a-istiod-ca-cert" pod "b255f882-402f-4db2-900b-051813a4300a" (UID: "b255f882-402f-4db2-900b-051813a4300a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:12:25.433352 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.416348    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b255f882-402f-4db2-900b-051813a4300a-kube-api-access-lsl7j" pod "b255f882-402f-4db2-900b-051813a4300a" (UID: "b255f882-402f-4db2-900b-051813a4300a"). InnerVolumeSpecName "kube-api-access-lsl7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:12:25.433352 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.431948    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"]
Oct 31 02:12:25.435278 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.435240    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-workload-certs" pod "b255f882-402f-4db2-900b-051813a4300a" (UID: "b255f882-402f-4db2-900b-051813a4300a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:12:25.435838 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.435754    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-istio-envoy" pod "b255f882-402f-4db2-900b-051813a4300a" (UID: "b255f882-402f-4db2-900b-051813a4300a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:12:25.440308 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.440274    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-cw4gg"]
Oct 31 02:12:25.444292 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.444268    5650 scope.go:122] "RemoveContainer" containerID="59dc66fa26cf1f535189f4243a938a8ae24805dfaa3854719c2693b9ee087812"
Oct 31 02:12:25.479921 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.479893    5650 scope.go:122] "RemoveContainer" containerID="6e5dbee1ee1c45eeec16a37cbd6b37e6d23d34d2028ec5d2dead95bf961e9031"
Oct 31 02:12:25.480549 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:12:25.480516    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e5dbee1ee1c45eeec16a37cbd6b37e6d23d34d2028ec5d2dead95bf961e9031\": container with ID starting with 6e5dbee1ee1c45eeec16a37cbd6b37e6d23d34d2028ec5d2dead95bf961e9031 not found: ID does not exist" containerID="6e5dbee1ee1c45eeec16a37cbd6b37e6d23d34d2028ec5d2dead95bf961e9031"
Oct 31 02:12:25.480681 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.480556    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e5dbee1ee1c45eeec16a37cbd6b37e6d23d34d2028ec5d2dead95bf961e9031"} err="failed to get container status \"6e5dbee1ee1c45eeec16a37cbd6b37e6d23d34d2028ec5d2dead95bf961e9031\": rpc error: code = NotFound desc = could not find container \"6e5dbee1ee1c45eeec16a37cbd6b37e6d23d34d2028ec5d2dead95bf961e9031\": container with ID starting with 6e5dbee1ee1c45eeec16a37cbd6b37e6d23d34d2028ec5d2dead95bf961e9031 not found: ID does not exist"
Oct 31 02:12:25.480681 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.480580    5650 scope.go:122] "RemoveContainer" containerID="59dc66fa26cf1f535189f4243a938a8ae24805dfaa3854719c2693b9ee087812"
Oct 31 02:12:25.480914 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:12:25.480893    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59dc66fa26cf1f535189f4243a938a8ae24805dfaa3854719c2693b9ee087812\": container with ID starting with 59dc66fa26cf1f535189f4243a938a8ae24805dfaa3854719c2693b9ee087812 not found: ID does not exist" containerID="59dc66fa26cf1f535189f4243a938a8ae24805dfaa3854719c2693b9ee087812"
Oct 31 02:12:25.480974 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.480927    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59dc66fa26cf1f535189f4243a938a8ae24805dfaa3854719c2693b9ee087812"} err="failed to get container status \"59dc66fa26cf1f535189f4243a938a8ae24805dfaa3854719c2693b9ee087812\": rpc error: code = NotFound desc = could not find container \"59dc66fa26cf1f535189f4243a938a8ae24805dfaa3854719c2693b9ee087812\": container with ID starting with 59dc66fa26cf1f535189f4243a938a8ae24805dfaa3854719c2693b9ee087812 not found: ID does not exist"
Oct 31 02:12:25.480974 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.480947    5650 scope.go:122] "RemoveContainer" containerID="6c36837539e4734e79cd5d6a380cf496e212645a8475ef34a23acdb876da0058"
Oct 31 02:12:25.495360 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.495184    5650 scope.go:122] "RemoveContainer" containerID="6c36837539e4734e79cd5d6a380cf496e212645a8475ef34a23acdb876da0058"
Oct 31 02:12:25.495672 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:12:25.495641    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c36837539e4734e79cd5d6a380cf496e212645a8475ef34a23acdb876da0058\": container with ID starting with 6c36837539e4734e79cd5d6a380cf496e212645a8475ef34a23acdb876da0058 not found: ID does not exist" containerID="6c36837539e4734e79cd5d6a380cf496e212645a8475ef34a23acdb876da0058"
Oct 31 02:12:25.495746 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.495686    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c36837539e4734e79cd5d6a380cf496e212645a8475ef34a23acdb876da0058"} err="failed to get container status \"6c36837539e4734e79cd5d6a380cf496e212645a8475ef34a23acdb876da0058\": rpc error: code = NotFound desc = could not find container \"6c36837539e4734e79cd5d6a380cf496e212645a8475ef34a23acdb876da0058\": container with ID starting with 6c36837539e4734e79cd5d6a380cf496e212645a8475ef34a23acdb876da0058 not found: ID does not exist"
Oct 31 02:12:25.504367 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.504330    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.504605 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.504381    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.504605 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.504403    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.504605 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.504421    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.504605 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.504502    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn98f\" (UniqueName: \"kubernetes.io/projected/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-kube-api-access-jn98f\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.504605 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.504546    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.504922 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.504698    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.504922 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.504726    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.504922 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.504773    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.504922 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.504870    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.505139 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.504923    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.505139 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.505015    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.505139 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.505036    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b255f882-402f-4db2-900b-051813a4300a-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.505139 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.505058    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b255f882-402f-4db2-900b-051813a4300a-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.505139 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.505074    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsl7j\" (UniqueName: \"kubernetes.io/projected/b255f882-402f-4db2-900b-051813a4300a-kube-api-access-lsl7j\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.505139 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.505091    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b255f882-402f-4db2-900b-051813a4300a-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.505139 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.505104    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.505139 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.505118    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.505139 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.505132    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.505534 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.505146    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b255f882-402f-4db2-900b-051813a4300a-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:25.505534 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.505192    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.505534 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.505498    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.507178 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.507155    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.507328 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.507309    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.518038 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.518006    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.518155 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.518103    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn98f\" (UniqueName: \"kubernetes.io/projected/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-kube-api-access-jn98f\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.629374 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.629332    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:25.746946 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.746902    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"]
Oct 31 02:12:25.750322 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.750287    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-2186c3-openshift-default-7f4f7d8dbf-w7w98"]
Oct 31 02:12:25.932725 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:12:25.932681    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60078b8_11e2_49a7_bbdb_c8a9ddfbc85e.slice/crio-8dc0e5e768ae7de23051a3a8c26a44dacbc14d39e2dd70ba087df3689f5e9daa WatchSource:0}: Error finding container 8dc0e5e768ae7de23051a3a8c26a44dacbc14d39e2dd70ba087df3689f5e9daa: Status 404 returned error can't find the container with id 8dc0e5e768ae7de23051a3a8c26a44dacbc14d39e2dd70ba087df3689f5e9daa
Oct 31 02:12:25.934577 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.934550    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"]
Oct 31 02:12:25.937204 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.937164    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:12:25.937317 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.937256    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:12:25.937317 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:25.937304    5650 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 02:12:26.126688 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:26.126646    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b255f882-402f-4db2-900b-051813a4300a" path="/var/lib/kubelet/pods/b255f882-402f-4db2-900b-051813a4300a/volumes"
Oct 31 02:12:26.127557 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:26.127533    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd4d7677-6d53-42e9-8416-024a78dc3ff7" path="/var/lib/kubelet/pods/cd4d7677-6d53-42e9-8416-024a78dc3ff7/volumes"
Oct 31 02:12:26.394344 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:26.394302    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9" event={"ID":"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e","Type":"ContainerStarted","Data":"6b01790ea75718bb3ceee509d456ee73d37499f4cea19ca192a391fd77a128eb"}
Oct 31 02:12:26.394936 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:26.394362    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9" event={"ID":"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e","Type":"ContainerStarted","Data":"8dc0e5e768ae7de23051a3a8c26a44dacbc14d39e2dd70ba087df3689f5e9daa"}
Oct 31 02:12:26.425434 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:26.425365    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9" podStartSLOduration=454.425342525 podStartE2EDuration="7m34.425342525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:12:26.422199634 +0000 UTC m=+3102.425913927" watchObservedRunningTime="2026-10-31 02:12:26.425342525 +0000 UTC m=+3102.429056817"
Oct 31 02:12:26.629574 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:26.629530    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:27.631505 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:27.631465    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Oct 31 02:12:27.632011 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:27.631975    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9" podUID="e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Oct 31 02:12:28.588989 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:12:28.588947    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:12:28.630117 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:28.630066    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Oct 31 02:12:28.630301 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:28.630140    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9" podUID="e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Oct 31 02:12:29.630111 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:29.630070    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Oct 31 02:12:29.630553 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:29.630134    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9" podUID="e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Oct 31 02:12:30.034939 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:30.034903    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c9acb54-qfbsh"]
Oct 31 02:12:30.039869 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:30.039837    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c9acb54-qfbsh"]
Oct 31 02:12:30.138326 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:30.138291    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="093087c0-0ee3-43b6-8bb7-d81f24fc7d72" path="/var/lib/kubelet/pods/093087c0-0ee3-43b6-8bb7-d81f24fc7d72/volumes"
Oct 31 02:12:30.630116 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:30.630080    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Oct 31 02:12:30.630719 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:30.630146    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9" podUID="e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Oct 31 02:12:31.636753 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:31.630388    5650 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body=
Oct 31 02:12:31.636753 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:31.630482    5650 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9" podUID="e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused"
Oct 31 02:12:32.672009 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:32.671971    5650 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:32.672772 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:32.672742    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:32.674054 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:32.674022    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:12:41.383351 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:41.383312    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k"]
Oct 31 02:12:41.439305 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:41.439234    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k" podUID="149232be-82fa-4fba-8a4f-dba80871843a" containerName="maas-api" containerID="cri-o://183feb5a08f976e420d32da3ef7dcfba6d8be0d80d6f99f6d1bfefd754426280" gracePeriod=30
Oct 31 02:12:42.163391 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:42.163346    5650 generic.go:356] "Generic (PLEG): container finished" podID="149232be-82fa-4fba-8a4f-dba80871843a" containerID="183feb5a08f976e420d32da3ef7dcfba6d8be0d80d6f99f6d1bfefd754426280" exitCode=0
Oct 31 02:12:42.164718 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:42.163826    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k" event={"ID":"149232be-82fa-4fba-8a4f-dba80871843a","Type":"ContainerDied","Data":"183feb5a08f976e420d32da3ef7dcfba6d8be0d80d6f99f6d1bfefd754426280"}
Oct 31 02:12:42.215265 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:42.215237    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k"
Oct 31 02:12:42.253242 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:42.253187    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/149232be-82fa-4fba-8a4f-dba80871843a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/149232be-82fa-4fba-8a4f-dba80871843a-openshift-service-ca\") pod \"149232be-82fa-4fba-8a4f-dba80871843a\" (UID: \"149232be-82fa-4fba-8a4f-dba80871843a\") "
Oct 31 02:12:42.254223 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:42.253359    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/149232be-82fa-4fba-8a4f-dba80871843a-kube-api-access-f2qn4\" (UniqueName: \"kubernetes.io/projected/149232be-82fa-4fba-8a4f-dba80871843a-kube-api-access-f2qn4\") pod \"149232be-82fa-4fba-8a4f-dba80871843a\" (UID: \"149232be-82fa-4fba-8a4f-dba80871843a\") "
Oct 31 02:12:42.254223 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:42.253789    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/149232be-82fa-4fba-8a4f-dba80871843a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/149232be-82fa-4fba-8a4f-dba80871843a-metrics-certs\") pod \"149232be-82fa-4fba-8a4f-dba80871843a\" (UID: \"149232be-82fa-4fba-8a4f-dba80871843a\") "
Oct 31 02:12:42.254223 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:42.253849    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/149232be-82fa-4fba-8a4f-dba80871843a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/149232be-82fa-4fba-8a4f-dba80871843a-maas-api-tls\") pod \"149232be-82fa-4fba-8a4f-dba80871843a\" (UID: \"149232be-82fa-4fba-8a4f-dba80871843a\") "
Oct 31 02:12:42.256244 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:42.256218    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/149232be-82fa-4fba-8a4f-dba80871843a-openshift-service-ca" pod "149232be-82fa-4fba-8a4f-dba80871843a" (UID: "149232be-82fa-4fba-8a4f-dba80871843a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:12:42.260641 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:42.260608    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/149232be-82fa-4fba-8a4f-dba80871843a-maas-api-tls" pod "149232be-82fa-4fba-8a4f-dba80871843a" (UID: "149232be-82fa-4fba-8a4f-dba80871843a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:12:42.260793 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:42.260638    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/149232be-82fa-4fba-8a4f-dba80871843a-metrics-certs" pod "149232be-82fa-4fba-8a4f-dba80871843a" (UID: "149232be-82fa-4fba-8a4f-dba80871843a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:12:42.260793 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:42.260689    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/149232be-82fa-4fba-8a4f-dba80871843a-kube-api-access-f2qn4" pod "149232be-82fa-4fba-8a4f-dba80871843a" (UID: "149232be-82fa-4fba-8a4f-dba80871843a"). InnerVolumeSpecName "kube-api-access-f2qn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:12:42.358954 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:42.358913    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2qn4\" (UniqueName: \"kubernetes.io/projected/149232be-82fa-4fba-8a4f-dba80871843a-kube-api-access-f2qn4\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:42.358954 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:42.358952    5650 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/149232be-82fa-4fba-8a4f-dba80871843a-metrics-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:42.358954 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:42.358965    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/149232be-82fa-4fba-8a4f-dba80871843a-maas-api-tls\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:42.360021 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:42.358975    5650 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/149232be-82fa-4fba-8a4f-dba80871843a-openshift-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:12:43.176513 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:43.176482    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k"
Oct 31 02:12:43.176513 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:43.176494    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k" event={"ID":"149232be-82fa-4fba-8a4f-dba80871843a","Type":"ContainerDied","Data":"da56da3b3d9358b6ba8164fd62388e981c7a7b1203c13b6ee0cf852fd2ccec35"}
Oct 31 02:12:43.177001 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:43.176559    5650 scope.go:122] "RemoveContainer" containerID="183feb5a08f976e420d32da3ef7dcfba6d8be0d80d6f99f6d1bfefd754426280"
Oct 31 02:12:43.213289 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:43.213244    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k"]
Oct 31 02:12:43.218134 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:43.218108    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-df9d42-59bc958db-85f5k"]
Oct 31 02:12:44.122561 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:12:44.122524    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="149232be-82fa-4fba-8a4f-dba80871843a" path="/var/lib/kubelet/pods/149232be-82fa-4fba-8a4f-dba80871843a/volumes"
Oct 31 02:12:58.555391 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:12:58.555346    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:13:08.222565 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:08.222519    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"]
Oct 31 02:13:08.245474 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:08.234729    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m" podUID="2cb82515-c5d2-47a6-af96-d0da5a75bcd3" containerName="istio-proxy" containerID="cri-o://2217942e8f57fd2623a321bd7bf5055a7ec4310bbd3a1aecf1a2f5d687f7c0ab" gracePeriod=30
Oct 31 02:13:08.249000 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:08.248962    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"]
Oct 31 02:13:08.250068 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:08.249189    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9" podUID="e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e" containerName="istio-proxy" containerID="cri-o://6b01790ea75718bb3ceee509d456ee73d37499f4cea19ca192a391fd77a128eb" gracePeriod=30
Oct 31 02:13:10.158022 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:10.157983    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"]
Oct 31 02:13:10.158426 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:10.158246    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8" podUID="fe7e7396-a227-4486-97ba-8b48945cc1d3" containerName="istio-proxy" containerID="cri-o://ada616721ed81bea6d5fcf304f4443e6241ffaa0ff2dc5d4d8a16783a9f13d59" gracePeriod=30
Oct 31 02:13:13.990907 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:13.990634    5650 generic.go:356] "Generic (PLEG): container finished" podID="e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e" containerID="6b01790ea75718bb3ceee509d456ee73d37499f4cea19ca192a391fd77a128eb" exitCode=0
Oct 31 02:13:13.991291 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:13.990996    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9" event={"ID":"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e","Type":"ContainerDied","Data":"6b01790ea75718bb3ceee509d456ee73d37499f4cea19ca192a391fd77a128eb"}
Oct 31 02:13:13.993805 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:13.993766    5650 generic.go:356] "Generic (PLEG): container finished" podID="2cb82515-c5d2-47a6-af96-d0da5a75bcd3" containerID="2217942e8f57fd2623a321bd7bf5055a7ec4310bbd3a1aecf1a2f5d687f7c0ab" exitCode=0
Oct 31 02:13:13.993968 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:13.993820    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m" event={"ID":"2cb82515-c5d2-47a6-af96-d0da5a75bcd3","Type":"ContainerDied","Data":"2217942e8f57fd2623a321bd7bf5055a7ec4310bbd3a1aecf1a2f5d687f7c0ab"}
Oct 31 02:13:13.993968 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:13.993880    5650 scope.go:122] "RemoveContainer" containerID="722b81cc31875b9ef514423be1993eaa0c5106f7b32341981e17336b1ec009f8"
Oct 31 02:13:14.055336 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.055309    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:13:14.094985 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.094953    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:13:14.116359 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.116324    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-podinfo\") pod \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") "
Oct 31 02:13:14.116587 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.116373    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-workload-socket\") pod \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") "
Oct 31 02:13:14.116587 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.116435    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-data\") pod \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") "
Oct 31 02:13:14.116742 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.116588    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-kube-api-access-b62lh\" (UniqueName: \"kubernetes.io/projected/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-kube-api-access-b62lh\") pod \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") "
Oct 31 02:13:14.116742 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.116660    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istiod-ca-cert\") pod \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") "
Oct 31 02:13:14.116867 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.116752    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-token\" (UniqueName: \"kubernetes.io/projected/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-token\") pod \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") "
Oct 31 02:13:14.116867 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.116806    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-data" pod "2cb82515-c5d2-47a6-af96-d0da5a75bcd3" (UID: "2cb82515-c5d2-47a6-af96-d0da5a75bcd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:14.117674 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.116885    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-credential-socket\") pod \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") "
Oct 31 02:13:14.117674 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.117041    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-envoy\") pod \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") "
Oct 31 02:13:14.117674 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.117045    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istiod-ca-cert" pod "2cb82515-c5d2-47a6-af96-d0da5a75bcd3" (UID: "2cb82515-c5d2-47a6-af96-d0da5a75bcd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:14.117674 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.117089    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-workload-socket" pod "2cb82515-c5d2-47a6-af96-d0da5a75bcd3" (UID: "2cb82515-c5d2-47a6-af96-d0da5a75bcd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:14.117674 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.117218    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-workload-certs\") pod \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\" (UID: \"2cb82515-c5d2-47a6-af96-d0da5a75bcd3\") "
Oct 31 02:13:14.117674 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.117601    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-workload-certs" pod "2cb82515-c5d2-47a6-af96-d0da5a75bcd3" (UID: "2cb82515-c5d2-47a6-af96-d0da5a75bcd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:14.117674 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.117636    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-credential-socket" pod "2cb82515-c5d2-47a6-af96-d0da5a75bcd3" (UID: "2cb82515-c5d2-47a6-af96-d0da5a75bcd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:14.118669 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.118098    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:14.118669 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.118113    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:14.118669 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.118127    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:14.118669 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.118138    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:14.118669 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.118154    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:14.119090 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.119064    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-podinfo" pod "2cb82515-c5d2-47a6-af96-d0da5a75bcd3" (UID: "2cb82515-c5d2-47a6-af96-d0da5a75bcd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:13:14.119888 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.119865    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-token" pod "2cb82515-c5d2-47a6-af96-d0da5a75bcd3" (UID: "2cb82515-c5d2-47a6-af96-d0da5a75bcd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:14.120028 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.120001    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-kube-api-access-b62lh" pod "2cb82515-c5d2-47a6-af96-d0da5a75bcd3" (UID: "2cb82515-c5d2-47a6-af96-d0da5a75bcd3"). InnerVolumeSpecName "kube-api-access-b62lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:14.121273 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.121224    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-envoy" pod "2cb82515-c5d2-47a6-af96-d0da5a75bcd3" (UID: "2cb82515-c5d2-47a6-af96-d0da5a75bcd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:14.219460 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.219412    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-envoy\") pod \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") "
Oct 31 02:13:14.219655 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.219559    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-kube-api-access-jn98f\" (UniqueName: \"kubernetes.io/projected/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-kube-api-access-jn98f\") pod \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") "
Oct 31 02:13:14.219712 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.219682    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-workload-certs\") pod \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") "
Oct 31 02:13:14.220440 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.219774    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-credential-socket\") pod \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") "
Oct 31 02:13:14.220440 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.219857    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-token\" (UniqueName: \"kubernetes.io/projected/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-token\") pod \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") "
Oct 31 02:13:14.220440 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.219880    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istiod-ca-cert\") pod \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") "
Oct 31 02:13:14.220440 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.219936    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-podinfo\") pod \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") "
Oct 31 02:13:14.220440 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.220017    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-workload-socket\") pod \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") "
Oct 31 02:13:14.220440 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.220016    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-workload-certs" pod "e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e" (UID: "e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:14.220440 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.220121    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-data\") pod \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\" (UID: \"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e\") "
Oct 31 02:13:14.220963 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.220587    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-credential-socket" pod "e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e" (UID: "e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:14.221676 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.221485    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-workload-socket" pod "e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e" (UID: "e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:14.221676 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.221522    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istiod-ca-cert" pod "e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e" (UID: "e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:14.222129 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.222018    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:14.222129 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.222043    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:14.222129 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.222059    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:14.222129 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.222074    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b62lh\" (UniqueName: \"kubernetes.io/projected/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-kube-api-access-b62lh\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:14.222129 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.222089    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:14.222129 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.222102    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb82515-c5d2-47a6-af96-d0da5a75bcd3-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:14.222129 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.222117    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:14.222129 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.222132    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:14.222612 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.222159    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-data" pod "e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e" (UID: "e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:14.222612 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.222231    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-kube-api-access-jn98f" pod "e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e" (UID: "e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e"). InnerVolumeSpecName "kube-api-access-jn98f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:14.223368 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.223344    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-token" pod "e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e" (UID: "e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:14.223535 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.223382    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-envoy" pod "e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e" (UID: "e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:14.224617 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.224595    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-podinfo" pod "e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e" (UID: "e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:13:14.324715 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.324677    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:14.324715 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.324713    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:14.324927 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.324731    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:14.324927 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.324747    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:14.324927 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:14.324760    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn98f\" (UniqueName: \"kubernetes.io/projected/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e-kube-api-access-jn98f\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:15.008078 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.008043    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"
Oct 31 02:13:15.008644 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.008072    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m" event={"ID":"2cb82515-c5d2-47a6-af96-d0da5a75bcd3","Type":"ContainerDied","Data":"d6f070103fbe1062288de64cb7fdd768aa5dd9978f96588c2bce9e8a6ca9c799"}
Oct 31 02:13:15.008644 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.008136    5650 scope.go:122] "RemoveContainer" containerID="2217942e8f57fd2623a321bd7bf5055a7ec4310bbd3a1aecf1a2f5d687f7c0ab"
Oct 31 02:13:15.010503 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.010411    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"
Oct 31 02:13:15.010503 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.010479    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9" event={"ID":"e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e","Type":"ContainerDied","Data":"8dc0e5e768ae7de23051a3a8c26a44dacbc14d39e2dd70ba087df3689f5e9daa"}
Oct 31 02:13:15.023436 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.023414    5650 scope.go:122] "RemoveContainer" containerID="6b01790ea75718bb3ceee509d456ee73d37499f4cea19ca192a391fd77a128eb"
Oct 31 02:13:15.058670 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.058633    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"]
Oct 31 02:13:15.063976 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.063944    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-nvtp9"]
Oct 31 02:13:15.076070 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.076040    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"]
Oct 31 02:13:15.086403 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.086363    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-pr62m"]
Oct 31 02:13:15.476416 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.476384    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 02:13:15.547579 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.547473    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-envoy\") pod \"fe7e7396-a227-4486-97ba-8b48945cc1d3\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") "
Oct 31 02:13:15.547764 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.547653    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-workload-certs\") pod \"fe7e7396-a227-4486-97ba-8b48945cc1d3\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") "
Oct 31 02:13:15.547764 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.547715    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-token\" (UniqueName: \"kubernetes.io/projected/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-token\") pod \"fe7e7396-a227-4486-97ba-8b48945cc1d3\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") "
Oct 31 02:13:15.547930 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.547822    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-data\") pod \"fe7e7396-a227-4486-97ba-8b48945cc1d3\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") "
Oct 31 02:13:15.547993 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.547974    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-workload-certs" pod "fe7e7396-a227-4486-97ba-8b48945cc1d3" (UID: "fe7e7396-a227-4486-97ba-8b48945cc1d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:15.548049 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.547979    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe7e7396-a227-4486-97ba-8b48945cc1d3-kube-api-access-5d2pd\" (UniqueName: \"kubernetes.io/projected/fe7e7396-a227-4486-97ba-8b48945cc1d3-kube-api-access-5d2pd\") pod \"fe7e7396-a227-4486-97ba-8b48945cc1d3\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") "
Oct 31 02:13:15.548114 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.548047    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-credential-socket\") pod \"fe7e7396-a227-4486-97ba-8b48945cc1d3\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") "
Oct 31 02:13:15.548114 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.548075    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe7e7396-a227-4486-97ba-8b48945cc1d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe7e7396-a227-4486-97ba-8b48945cc1d3-istiod-ca-cert\") pod \"fe7e7396-a227-4486-97ba-8b48945cc1d3\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") "
Oct 31 02:13:15.548236 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.548119    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-workload-socket\") pod \"fe7e7396-a227-4486-97ba-8b48945cc1d3\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") "
Oct 31 02:13:15.548302 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.548260    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-podinfo\") pod \"fe7e7396-a227-4486-97ba-8b48945cc1d3\" (UID: \"fe7e7396-a227-4486-97ba-8b48945cc1d3\") "
Oct 31 02:13:15.548901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.548375    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-credential-socket" pod "fe7e7396-a227-4486-97ba-8b48945cc1d3" (UID: "fe7e7396-a227-4486-97ba-8b48945cc1d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:15.548901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.548556    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-workload-socket" pod "fe7e7396-a227-4486-97ba-8b48945cc1d3" (UID: "fe7e7396-a227-4486-97ba-8b48945cc1d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:15.548901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.548631    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-data" pod "fe7e7396-a227-4486-97ba-8b48945cc1d3" (UID: "fe7e7396-a227-4486-97ba-8b48945cc1d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:15.549233 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.549209    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe7e7396-a227-4486-97ba-8b48945cc1d3-istiod-ca-cert" pod "fe7e7396-a227-4486-97ba-8b48945cc1d3" (UID: "fe7e7396-a227-4486-97ba-8b48945cc1d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:15.549382 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.549355    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:15.549382 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.549377    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:15.549578 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.549391    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:15.549578 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.549404    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:15.549578 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.549417    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe7e7396-a227-4486-97ba-8b48945cc1d3-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:15.551415 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.551387    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-envoy" pod "fe7e7396-a227-4486-97ba-8b48945cc1d3" (UID: "fe7e7396-a227-4486-97ba-8b48945cc1d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:15.551541 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.551419    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-token" pod "fe7e7396-a227-4486-97ba-8b48945cc1d3" (UID: "fe7e7396-a227-4486-97ba-8b48945cc1d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:15.551541 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.551483    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe7e7396-a227-4486-97ba-8b48945cc1d3-kube-api-access-5d2pd" pod "fe7e7396-a227-4486-97ba-8b48945cc1d3" (UID: "fe7e7396-a227-4486-97ba-8b48945cc1d3"). InnerVolumeSpecName "kube-api-access-5d2pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:15.551766 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.551737    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-podinfo" pod "fe7e7396-a227-4486-97ba-8b48945cc1d3" (UID: "fe7e7396-a227-4486-97ba-8b48945cc1d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:13:15.651100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.651058    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d2pd\" (UniqueName: \"kubernetes.io/projected/fe7e7396-a227-4486-97ba-8b48945cc1d3-kube-api-access-5d2pd\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:15.651100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.651097    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:15.651100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.651107    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:15.651356 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:15.651116    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe7e7396-a227-4486-97ba-8b48945cc1d3-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:16.039545 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:16.039505    5650 generic.go:356] "Generic (PLEG): container finished" podID="fe7e7396-a227-4486-97ba-8b48945cc1d3" containerID="ada616721ed81bea6d5fcf304f4443e6241ffaa0ff2dc5d4d8a16783a9f13d59" exitCode=0
Oct 31 02:13:16.040047 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:16.039584    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"
Oct 31 02:13:16.040047 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:16.039588    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8" event={"ID":"fe7e7396-a227-4486-97ba-8b48945cc1d3","Type":"ContainerDied","Data":"ada616721ed81bea6d5fcf304f4443e6241ffaa0ff2dc5d4d8a16783a9f13d59"}
Oct 31 02:13:16.040047 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:16.039652    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8" event={"ID":"fe7e7396-a227-4486-97ba-8b48945cc1d3","Type":"ContainerDied","Data":"cafa78e9ce3e140ddcf83c6c0a7c2aacc57f7d9983cb84ef8bf84934adfaf6de"}
Oct 31 02:13:16.040047 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:16.039688    5650 scope.go:122] "RemoveContainer" containerID="ada616721ed81bea6d5fcf304f4443e6241ffaa0ff2dc5d4d8a16783a9f13d59"
Oct 31 02:13:16.056896 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:16.056874    5650 scope.go:122] "RemoveContainer" containerID="273ebfeeb08e9008b5e71fc27415a534e938057c3de950849f7161c7bc6f14d8"
Oct 31 02:13:16.070729 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:16.070678    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"]
Oct 31 02:13:16.072102 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:16.072081    5650 scope.go:122] "RemoveContainer" containerID="ada616721ed81bea6d5fcf304f4443e6241ffaa0ff2dc5d4d8a16783a9f13d59"
Oct 31 02:13:16.072507 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:13:16.072436    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ada616721ed81bea6d5fcf304f4443e6241ffaa0ff2dc5d4d8a16783a9f13d59\": container with ID starting with ada616721ed81bea6d5fcf304f4443e6241ffaa0ff2dc5d4d8a16783a9f13d59 not found: ID does not exist" containerID="ada616721ed81bea6d5fcf304f4443e6241ffaa0ff2dc5d4d8a16783a9f13d59"
Oct 31 02:13:16.072623 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:16.072513    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ada616721ed81bea6d5fcf304f4443e6241ffaa0ff2dc5d4d8a16783a9f13d59"} err="failed to get container status \"ada616721ed81bea6d5fcf304f4443e6241ffaa0ff2dc5d4d8a16783a9f13d59\": rpc error: code = NotFound desc = could not find container \"ada616721ed81bea6d5fcf304f4443e6241ffaa0ff2dc5d4d8a16783a9f13d59\": container with ID starting with ada616721ed81bea6d5fcf304f4443e6241ffaa0ff2dc5d4d8a16783a9f13d59 not found: ID does not exist"
Oct 31 02:13:16.072623 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:16.072533    5650 scope.go:122] "RemoveContainer" containerID="273ebfeeb08e9008b5e71fc27415a534e938057c3de950849f7161c7bc6f14d8"
Oct 31 02:13:16.072843 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:13:16.072823    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273ebfeeb08e9008b5e71fc27415a534e938057c3de950849f7161c7bc6f14d8\": container with ID starting with 273ebfeeb08e9008b5e71fc27415a534e938057c3de950849f7161c7bc6f14d8 not found: ID does not exist" containerID="273ebfeeb08e9008b5e71fc27415a534e938057c3de950849f7161c7bc6f14d8"
Oct 31 02:13:16.072922 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:16.072851    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273ebfeeb08e9008b5e71fc27415a534e938057c3de950849f7161c7bc6f14d8"} err="failed to get container status \"273ebfeeb08e9008b5e71fc27415a534e938057c3de950849f7161c7bc6f14d8\": rpc error: code = NotFound desc = could not find container \"273ebfeeb08e9008b5e71fc27415a534e938057c3de950849f7161c7bc6f14d8\": container with ID starting with 273ebfeeb08e9008b5e71fc27415a534e938057c3de950849f7161c7bc6f14d8 not found: ID does not exist"
Oct 31 02:13:16.077268 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:16.077239    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-vxzf8"]
Oct 31 02:13:16.121211 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:16.121158    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cb82515-c5d2-47a6-af96-d0da5a75bcd3" path="/var/lib/kubelet/pods/2cb82515-c5d2-47a6-af96-d0da5a75bcd3/volumes"
Oct 31 02:13:16.122008 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:16.121982    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e" path="/var/lib/kubelet/pods/e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e/volumes"
Oct 31 02:13:16.122694 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:16.122671    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe7e7396-a227-4486-97ba-8b48945cc1d3" path="/var/lib/kubelet/pods/fe7e7396-a227-4486-97ba-8b48945cc1d3/volumes"
Oct 31 02:13:19.753400 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:19.753362    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"]
Oct 31 02:13:19.753967 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:19.753652    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj" podUID="21d3adaa-ed6d-46d1-8990-62b11d6d0adc" containerName="istio-proxy" containerID="cri-o://cb1dd73110454f5319ed9a2801e2d8d1c6603f0813c77b8d4030f88e5c51ecf3" gracePeriod=30
Oct 31 02:13:25.154424 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.154287    5650 generic.go:356] "Generic (PLEG): container finished" podID="21d3adaa-ed6d-46d1-8990-62b11d6d0adc" containerID="cb1dd73110454f5319ed9a2801e2d8d1c6603f0813c77b8d4030f88e5c51ecf3" exitCode=0
Oct 31 02:13:25.154876 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.154588    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj" event={"ID":"21d3adaa-ed6d-46d1-8990-62b11d6d0adc","Type":"ContainerDied","Data":"cb1dd73110454f5319ed9a2801e2d8d1c6603f0813c77b8d4030f88e5c51ecf3"}
Oct 31 02:13:25.596645 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.596612    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:13:25.618069 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.618027    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-kube-api-access-5zk22\" (UniqueName: \"kubernetes.io/projected/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-kube-api-access-5zk22\") pod \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") "
Oct 31 02:13:25.618250 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.618103    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-workload-certs\") pod \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") "
Oct 31 02:13:25.618250 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.618160    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-data\") pod \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") "
Oct 31 02:13:25.618250 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.618209    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-credential-socket\") pod \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") "
Oct 31 02:13:25.618419 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.618397    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-token\" (UniqueName: \"kubernetes.io/projected/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-token\") pod \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") "
Oct 31 02:13:25.618555 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.618418    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-workload-certs" pod "21d3adaa-ed6d-46d1-8990-62b11d6d0adc" (UID: "21d3adaa-ed6d-46d1-8990-62b11d6d0adc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:25.618555 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.618547    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-workload-socket\") pod \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") "
Oct 31 02:13:25.618707 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.618646    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-podinfo\") pod \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") "
Oct 31 02:13:25.618815 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.618795    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istiod-ca-cert\") pod \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") "
Oct 31 02:13:25.618927 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.618852    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-envoy\") pod \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\" (UID: \"21d3adaa-ed6d-46d1-8990-62b11d6d0adc\") "
Oct 31 02:13:25.618927 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.618795    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-credential-socket" pod "21d3adaa-ed6d-46d1-8990-62b11d6d0adc" (UID: "21d3adaa-ed6d-46d1-8990-62b11d6d0adc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:25.619108 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.618850    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-workload-socket" pod "21d3adaa-ed6d-46d1-8990-62b11d6d0adc" (UID: "21d3adaa-ed6d-46d1-8990-62b11d6d0adc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:25.619214 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.619141    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-data" pod "21d3adaa-ed6d-46d1-8990-62b11d6d0adc" (UID: "21d3adaa-ed6d-46d1-8990-62b11d6d0adc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:25.619763 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.619742    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:25.619879 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.619766    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:25.619879 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.619780    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:25.619879 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.619795    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:25.621081 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.620566    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istiod-ca-cert" pod "21d3adaa-ed6d-46d1-8990-62b11d6d0adc" (UID: "21d3adaa-ed6d-46d1-8990-62b11d6d0adc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:25.621081 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.620966    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-kube-api-access-5zk22" pod "21d3adaa-ed6d-46d1-8990-62b11d6d0adc" (UID: "21d3adaa-ed6d-46d1-8990-62b11d6d0adc"). InnerVolumeSpecName "kube-api-access-5zk22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:25.621671 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.621645    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-podinfo" pod "21d3adaa-ed6d-46d1-8990-62b11d6d0adc" (UID: "21d3adaa-ed6d-46d1-8990-62b11d6d0adc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:13:25.622255 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.622227    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-envoy" pod "21d3adaa-ed6d-46d1-8990-62b11d6d0adc" (UID: "21d3adaa-ed6d-46d1-8990-62b11d6d0adc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:25.622843 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.622819    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-token" pod "21d3adaa-ed6d-46d1-8990-62b11d6d0adc" (UID: "21d3adaa-ed6d-46d1-8990-62b11d6d0adc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:25.722019 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.721977    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:25.722019 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.722020    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:25.722019 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.722038    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:25.722417 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.722053    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:25.722417 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:25.722071    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zk22\" (UniqueName: \"kubernetes.io/projected/21d3adaa-ed6d-46d1-8990-62b11d6d0adc-kube-api-access-5zk22\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:26.165008 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:26.164967    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"
Oct 31 02:13:26.165008 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:26.164976    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj" event={"ID":"21d3adaa-ed6d-46d1-8990-62b11d6d0adc","Type":"ContainerDied","Data":"f4c5b509836bfec59f4740d54718614cdca3800773ee82108e3670eca9f25c5a"}
Oct 31 02:13:26.165632 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:26.165046    5650 scope.go:122] "RemoveContainer" containerID="cb1dd73110454f5319ed9a2801e2d8d1c6603f0813c77b8d4030f88e5c51ecf3"
Oct 31 02:13:26.193140 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:26.193101    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"]
Oct 31 02:13:26.199894 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:26.199866    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-ljvtj"]
Oct 31 02:13:28.121280 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:28.121245    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21d3adaa-ed6d-46d1-8990-62b11d6d0adc" path="/var/lib/kubelet/pods/21d3adaa-ed6d-46d1-8990-62b11d6d0adc/volumes"
Oct 31 02:13:28.561980 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:13:28.561922    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:13:32.630142 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:32.630106    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"]
Oct 31 02:13:32.630674 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:32.630310    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8" podUID="247b6d6e-faaf-44e3-bf73-73fbb620f5e0" containerName="istio-proxy" containerID="cri-o://0763e56a16c51dd04345c4a2ab527e5d949a68d26a69689278349dcd65b672c3" gracePeriod=30
Oct 31 02:13:35.625382 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:35.599695    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:13:35.706372 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:35.706073    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8" podUID="247b6d6e-faaf-44e3-bf73-73fbb620f5e0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:13:37.406008 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.405419    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39c920b7-6kchh"]
Oct 31 02:13:37.406766 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.406736    5650 cpu_manager.go:414] "containerMap: removing container" podUID="b255f882-402f-4db2-900b-051813a4300a" containerName="istio-proxy"
Oct 31 02:13:37.406947 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.406921    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b255f882-402f-4db2-900b-051813a4300a" containerName="istio-proxy"
Oct 31 02:13:37.407061 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.406991    5650 cpu_manager.go:414] "containerMap: removing container" podUID="2cb82515-c5d2-47a6-af96-d0da5a75bcd3" containerName="istio-proxy"
Oct 31 02:13:37.407061 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.407000    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb82515-c5d2-47a6-af96-d0da5a75bcd3" containerName="istio-proxy"
Oct 31 02:13:37.407061 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.407014    5650 cpu_manager.go:414] "containerMap: removing container" podUID="21d3adaa-ed6d-46d1-8990-62b11d6d0adc" containerName="istio-proxy"
Oct 31 02:13:37.407061 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.407021    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d3adaa-ed6d-46d1-8990-62b11d6d0adc" containerName="istio-proxy"
Oct 31 02:13:37.407061 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.407042    5650 cpu_manager.go:414] "containerMap: removing container" podUID="fe7e7396-a227-4486-97ba-8b48945cc1d3" containerName="istio-proxy"
Oct 31 02:13:37.407061 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.407057    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe7e7396-a227-4486-97ba-8b48945cc1d3" containerName="istio-proxy"
Oct 31 02:13:37.407490 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.407103    5650 cpu_manager.go:414] "containerMap: removing container" podUID="149232be-82fa-4fba-8a4f-dba80871843a" containerName="maas-api"
Oct 31 02:13:37.407490 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.407110    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="149232be-82fa-4fba-8a4f-dba80871843a" containerName="maas-api"
Oct 31 02:13:37.407490 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.407139    5650 cpu_manager.go:414] "containerMap: removing container" podUID="fe7e7396-a227-4486-97ba-8b48945cc1d3" containerName="istio-proxy"
Oct 31 02:13:37.407490 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.407146    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe7e7396-a227-4486-97ba-8b48945cc1d3" containerName="istio-proxy"
Oct 31 02:13:37.407490 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.407159    5650 cpu_manager.go:414] "containerMap: removing container" podUID="e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e" containerName="istio-proxy"
Oct 31 02:13:37.407490 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.407165    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e" containerName="istio-proxy"
Oct 31 02:13:37.407742 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.407568    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="b255f882-402f-4db2-900b-051813a4300a" containerName="istio-proxy"
Oct 31 02:13:37.407742 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.407592    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb82515-c5d2-47a6-af96-d0da5a75bcd3" containerName="istio-proxy"
Oct 31 02:13:37.407742 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.407601    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="21d3adaa-ed6d-46d1-8990-62b11d6d0adc" containerName="istio-proxy"
Oct 31 02:13:37.407742 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.407612    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe7e7396-a227-4486-97ba-8b48945cc1d3" containerName="istio-proxy"
Oct 31 02:13:37.407742 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.407630    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="149232be-82fa-4fba-8a4f-dba80871843a" containerName="maas-api"
Oct 31 02:13:37.407742 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.407643    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="e60078b8-11e2-49a7-bbdb-c8a9ddfbc85e" containerName="istio-proxy"
Oct 31 02:13:37.423269 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.423238    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39c920b7-6kchh"
Oct 31 02:13:37.425507 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.424688    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39c920b7-6kchh"]
Oct 31 02:13:37.428164 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.428139    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:13:37.430047 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.430022    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:13:37.589986 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.589940    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014b9592-996c-4034-8453-04e92db14801-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39c920b7-6kchh\" (UID: \"014b9592-996c-4034-8453-04e92db14801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39c920b7-6kchh"
Oct 31 02:13:37.590339 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.590314    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014b9592-996c-4034-8453-04e92db14801-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39c920b7-6kchh\" (UID: \"014b9592-996c-4034-8453-04e92db14801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39c920b7-6kchh"
Oct 31 02:13:37.590497 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.590477    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfnnr\" (UniqueName: \"kubernetes.io/projected/014b9592-996c-4034-8453-04e92db14801-kube-api-access-cfnnr\") pod \"maas-api-revoke-sub-39c920b7-6kchh\" (UID: \"014b9592-996c-4034-8453-04e92db14801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39c920b7-6kchh"
Oct 31 02:13:37.692184 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.692146    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014b9592-996c-4034-8453-04e92db14801-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39c920b7-6kchh\" (UID: \"014b9592-996c-4034-8453-04e92db14801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39c920b7-6kchh"
Oct 31 02:13:37.692400 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.692375    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014b9592-996c-4034-8453-04e92db14801-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39c920b7-6kchh\" (UID: \"014b9592-996c-4034-8453-04e92db14801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39c920b7-6kchh"
Oct 31 02:13:37.692515 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.692439    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfnnr\" (UniqueName: \"kubernetes.io/projected/014b9592-996c-4034-8453-04e92db14801-kube-api-access-cfnnr\") pod \"maas-api-revoke-sub-39c920b7-6kchh\" (UID: \"014b9592-996c-4034-8453-04e92db14801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39c920b7-6kchh"
Oct 31 02:13:37.692977 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.692944    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014b9592-996c-4034-8453-04e92db14801-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39c920b7-6kchh\" (UID: \"014b9592-996c-4034-8453-04e92db14801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39c920b7-6kchh"
Oct 31 02:13:37.694908 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.694883    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014b9592-996c-4034-8453-04e92db14801-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39c920b7-6kchh\" (UID: \"014b9592-996c-4034-8453-04e92db14801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39c920b7-6kchh"
Oct 31 02:13:37.706263 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.706236    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfnnr\" (UniqueName: \"kubernetes.io/projected/014b9592-996c-4034-8453-04e92db14801-kube-api-access-cfnnr\") pod \"maas-api-revoke-sub-39c920b7-6kchh\" (UID: \"014b9592-996c-4034-8453-04e92db14801\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39c920b7-6kchh"
Oct 31 02:13:37.757903 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:37.757875    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39c920b7-6kchh"
Oct 31 02:13:38.779327 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.779295    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 02:13:38.803342 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.803306    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"]
Oct 31 02:13:38.803497 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.803391    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"]
Oct 31 02:13:38.803497 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.803428    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39c920b7-6kchh"]
Oct 31 02:13:38.803701 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.803653    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m" podUID="bfa28245-e7ac-49b8-a507-4b452bb944e7" containerName="istio-proxy" containerID="cri-o://9a85ecca2d62a9abe6dbf01a8437fbebb8d3643bd1fd44f3b0ddca3adc05e2a4" gracePeriod=30
Oct 31 02:13:38.804038 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.803827    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np" podUID="0d39c057-51e1-49f1-85de-facd9e8f15fc" containerName="istio-proxy" containerID="cri-o://0f3f7de50e68596f8b21df5e920197215208339ecb95644d9fa3cd80501526e0" gracePeriod=30
Oct 31 02:13:38.806123 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.806093    5650 generic.go:356] "Generic (PLEG): container finished" podID="247b6d6e-faaf-44e3-bf73-73fbb620f5e0" containerID="0763e56a16c51dd04345c4a2ab527e5d949a68d26a69689278349dcd65b672c3" exitCode=0
Oct 31 02:13:38.806208 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.806159    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8" event={"ID":"247b6d6e-faaf-44e3-bf73-73fbb620f5e0","Type":"ContainerDied","Data":"0763e56a16c51dd04345c4a2ab527e5d949a68d26a69689278349dcd65b672c3"}
Oct 31 02:13:38.806258 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.806207    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8" event={"ID":"247b6d6e-faaf-44e3-bf73-73fbb620f5e0","Type":"ContainerDied","Data":"bfa1a2d1d2ca7d7dfd8a9d2b07adbab73196d864a8001b3e42d08a9197032b77"}
Oct 31 02:13:38.806258 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.806236    5650 scope.go:122] "RemoveContainer" containerID="0763e56a16c51dd04345c4a2ab527e5d949a68d26a69689278349dcd65b672c3"
Oct 31 02:13:38.806395 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.806382    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"
Oct 31 02:13:38.829162 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.829130    5650 scope.go:122] "RemoveContainer" containerID="c7c398c7b60890635bba29a9eb4f236353288f7452fc2dc48736905d4bb563b5"
Oct 31 02:13:38.846111 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.845747    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-token\" (UniqueName: \"kubernetes.io/projected/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-token\") pod \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") "
Oct 31 02:13:38.846111 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.845793    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-envoy\") pod \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") "
Oct 31 02:13:38.846111 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.845827    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-kube-api-access-gk5zv\" (UniqueName: \"kubernetes.io/projected/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-kube-api-access-gk5zv\") pod \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") "
Oct 31 02:13:38.846111 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.845992    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-workload-certs\") pod \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") "
Oct 31 02:13:38.846111 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.846094    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istiod-ca-cert\") pod \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") "
Oct 31 02:13:38.846630 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.846203    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-credential-socket\") pod \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") "
Oct 31 02:13:38.846630 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.846228    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-data\") pod \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") "
Oct 31 02:13:38.846630 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.846292    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-podinfo\") pod \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") "
Oct 31 02:13:38.846630 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.846330    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-workload-certs" pod "247b6d6e-faaf-44e3-bf73-73fbb620f5e0" (UID: "247b6d6e-faaf-44e3-bf73-73fbb620f5e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:38.846630 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.846536    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-workload-socket\") pod \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\" (UID: \"247b6d6e-faaf-44e3-bf73-73fbb620f5e0\") "
Oct 31 02:13:38.847038 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.847010    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-credential-socket" pod "247b6d6e-faaf-44e3-bf73-73fbb620f5e0" (UID: "247b6d6e-faaf-44e3-bf73-73fbb620f5e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:38.847325 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.847303    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-workload-socket" pod "247b6d6e-faaf-44e3-bf73-73fbb620f5e0" (UID: "247b6d6e-faaf-44e3-bf73-73fbb620f5e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:38.847666 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.847531    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-data" pod "247b6d6e-faaf-44e3-bf73-73fbb620f5e0" (UID: "247b6d6e-faaf-44e3-bf73-73fbb620f5e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:38.852306 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.848581    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:38.852306 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.848608    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:38.852306 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.848623    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:38.852306 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.848638    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:38.852306 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.850157    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istiod-ca-cert" pod "247b6d6e-faaf-44e3-bf73-73fbb620f5e0" (UID: "247b6d6e-faaf-44e3-bf73-73fbb620f5e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:38.852306 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.850293    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-envoy" pod "247b6d6e-faaf-44e3-bf73-73fbb620f5e0" (UID: "247b6d6e-faaf-44e3-bf73-73fbb620f5e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:38.852306 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.851555    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-podinfo" pod "247b6d6e-faaf-44e3-bf73-73fbb620f5e0" (UID: "247b6d6e-faaf-44e3-bf73-73fbb620f5e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:13:38.852306 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.852240    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-kube-api-access-gk5zv" pod "247b6d6e-faaf-44e3-bf73-73fbb620f5e0" (UID: "247b6d6e-faaf-44e3-bf73-73fbb620f5e0"). InnerVolumeSpecName "kube-api-access-gk5zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:38.868225 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.868190    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-token" pod "247b6d6e-faaf-44e3-bf73-73fbb620f5e0" (UID: "247b6d6e-faaf-44e3-bf73-73fbb620f5e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:38.921185 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.921131    5650 scope.go:122] "RemoveContainer" containerID="0763e56a16c51dd04345c4a2ab527e5d949a68d26a69689278349dcd65b672c3"
Oct 31 02:13:38.921594 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:13:38.921566    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0763e56a16c51dd04345c4a2ab527e5d949a68d26a69689278349dcd65b672c3\": container with ID starting with 0763e56a16c51dd04345c4a2ab527e5d949a68d26a69689278349dcd65b672c3 not found: ID does not exist" containerID="0763e56a16c51dd04345c4a2ab527e5d949a68d26a69689278349dcd65b672c3"
Oct 31 02:13:38.921707 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.921606    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0763e56a16c51dd04345c4a2ab527e5d949a68d26a69689278349dcd65b672c3"} err="failed to get container status \"0763e56a16c51dd04345c4a2ab527e5d949a68d26a69689278349dcd65b672c3\": rpc error: code = NotFound desc = could not find container \"0763e56a16c51dd04345c4a2ab527e5d949a68d26a69689278349dcd65b672c3\": container with ID starting with 0763e56a16c51dd04345c4a2ab527e5d949a68d26a69689278349dcd65b672c3 not found: ID does not exist"
Oct 31 02:13:38.921707 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.921629    5650 scope.go:122] "RemoveContainer" containerID="c7c398c7b60890635bba29a9eb4f236353288f7452fc2dc48736905d4bb563b5"
Oct 31 02:13:38.921944 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:13:38.921923    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7c398c7b60890635bba29a9eb4f236353288f7452fc2dc48736905d4bb563b5\": container with ID starting with c7c398c7b60890635bba29a9eb4f236353288f7452fc2dc48736905d4bb563b5 not found: ID does not exist" containerID="c7c398c7b60890635bba29a9eb4f236353288f7452fc2dc48736905d4bb563b5"
Oct 31 02:13:38.922007 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:38.921954    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c398c7b60890635bba29a9eb4f236353288f7452fc2dc48736905d4bb563b5"} err="failed to get container status \"c7c398c7b60890635bba29a9eb4f236353288f7452fc2dc48736905d4bb563b5\": rpc error: code = NotFound desc = could not find container \"c7c398c7b60890635bba29a9eb4f236353288f7452fc2dc48736905d4bb563b5\": container with ID starting with c7c398c7b60890635bba29a9eb4f236353288f7452fc2dc48736905d4bb563b5 not found: ID does not exist"
Oct 31 02:13:39.027396 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:39.019572    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:39.027396 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:39.019609    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:39.027396 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:39.019624    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:39.027396 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:39.019665    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:39.027396 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:39.019680    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk5zv\" (UniqueName: \"kubernetes.io/projected/247b6d6e-faaf-44e3-bf73-73fbb620f5e0-kube-api-access-gk5zv\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:39.138523 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:39.138483    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"]
Oct 31 02:13:39.144803 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:39.144776    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2e6bfd-openshift-default-5fd54bbdc5-6xff8"]
Oct 31 02:13:39.817232 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:39.817198    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39c920b7-6kchh" event={"ID":"014b9592-996c-4034-8453-04e92db14801","Type":"ContainerStarted","Data":"90eef6dc8c824bb334b0154a21eaa5728392d3b9d738c33426beb05137dd6cd8"}
Oct 31 02:13:39.819317 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:39.817252    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39c920b7-6kchh" event={"ID":"014b9592-996c-4034-8453-04e92db14801","Type":"ContainerStarted","Data":"83cd42e45a82f9d8c838e467603e16b79885061a928c94e7fd04dc487a8ff7c7"}
Oct 31 02:13:39.842868 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:39.842818    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39c920b7-6kchh" podStartSLOduration=2.842803183 podStartE2EDuration="2.842803183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:13:39.838359395 +0000 UTC m=+3175.842073686" watchObservedRunningTime="2026-10-31 02:13:39.842803183 +0000 UTC m=+3175.846517474"
Oct 31 02:13:40.126167 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:40.126124    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="247b6d6e-faaf-44e3-bf73-73fbb620f5e0" path="/var/lib/kubelet/pods/247b6d6e-faaf-44e3-bf73-73fbb620f5e0/volumes"
Oct 31 02:13:40.827546 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:40.827515    5650 generic.go:356] "Generic (PLEG): container finished" podID="014b9592-996c-4034-8453-04e92db14801" containerID="90eef6dc8c824bb334b0154a21eaa5728392d3b9d738c33426beb05137dd6cd8" exitCode=0
Oct 31 02:13:40.827974 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:40.827600    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39c920b7-6kchh" event={"ID":"014b9592-996c-4034-8453-04e92db14801","Type":"ContainerDied","Data":"90eef6dc8c824bb334b0154a21eaa5728392d3b9d738c33426beb05137dd6cd8"}
Oct 31 02:13:41.013192 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:41.013158    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"]
Oct 31 02:13:41.013383 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:41.013361    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr" podUID="e9c1e7ce-8191-44fd-a838-b9718752488a" containerName="istio-proxy" containerID="cri-o://2f5e6715f052080d3c5c0d7449e23e6ff65587e0ce755b1785ad3e9ca0e0f339" gracePeriod=30
Oct 31 02:13:42.019437 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:42.019397    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39c920b7-6kchh"
Oct 31 02:13:42.202702 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:42.202675    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/014b9592-996c-4034-8453-04e92db14801-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014b9592-996c-4034-8453-04e92db14801-maas-api-xks-ca\") pod \"014b9592-996c-4034-8453-04e92db14801\" (UID: \"014b9592-996c-4034-8453-04e92db14801\") "
Oct 31 02:13:42.202881 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:42.202719    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/014b9592-996c-4034-8453-04e92db14801-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014b9592-996c-4034-8453-04e92db14801-maas-api-openshift-ca\") pod \"014b9592-996c-4034-8453-04e92db14801\" (UID: \"014b9592-996c-4034-8453-04e92db14801\") "
Oct 31 02:13:42.202881 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:42.202740    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014b9592-996c-4034-8453-04e92db14801-kube-api-access-cfnnr\" (UniqueName: \"kubernetes.io/projected/014b9592-996c-4034-8453-04e92db14801-kube-api-access-cfnnr\") pod \"014b9592-996c-4034-8453-04e92db14801\" (UID: \"014b9592-996c-4034-8453-04e92db14801\") "
Oct 31 02:13:42.203138 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:42.203112    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014b9592-996c-4034-8453-04e92db14801-maas-api-openshift-ca" pod "014b9592-996c-4034-8453-04e92db14801" (UID: "014b9592-996c-4034-8453-04e92db14801"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:42.203842 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:42.203376    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014b9592-996c-4034-8453-04e92db14801-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:42.204929 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:42.204908    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014b9592-996c-4034-8453-04e92db14801-kube-api-access-cfnnr" pod "014b9592-996c-4034-8453-04e92db14801" (UID: "014b9592-996c-4034-8453-04e92db14801"). InnerVolumeSpecName "kube-api-access-cfnnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:42.205029 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:42.204950    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/014b9592-996c-4034-8453-04e92db14801-maas-api-xks-ca" pod "014b9592-996c-4034-8453-04e92db14801" (UID: "014b9592-996c-4034-8453-04e92db14801"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:13:42.306085 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:42.306004    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014b9592-996c-4034-8453-04e92db14801-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:42.306085 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:42.306083    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfnnr\" (UniqueName: \"kubernetes.io/projected/014b9592-996c-4034-8453-04e92db14801-kube-api-access-cfnnr\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:42.848784 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:42.848753    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39c920b7-6kchh"
Oct 31 02:13:42.849588 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:42.848787    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39c920b7-6kchh" event={"ID":"014b9592-996c-4034-8453-04e92db14801","Type":"ContainerDied","Data":"83cd42e45a82f9d8c838e467603e16b79885061a928c94e7fd04dc487a8ff7c7"}
Oct 31 02:13:42.849588 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:42.848841    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83cd42e45a82f9d8c838e467603e16b79885061a928c94e7fd04dc487a8ff7c7"
Oct 31 02:13:43.724800 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:43.724759    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"]
Oct 31 02:13:43.764015 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:43.763978    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"]
Oct 31 02:13:43.764267 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:43.764239    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2" podUID="214c7868-1e53-463f-8247-bed23603debd" containerName="istio-proxy" containerID="cri-o://a73b3a3e25c8fc6a3df26a76791f6ccf7ad5e2ed362aaefb3417df8b4f789cb2" gracePeriod=30
Oct 31 02:13:44.168347 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.168315    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 02:13:44.231740 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.231702    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:13:44.252340 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.252305    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-workload-socket\") pod \"bfa28245-e7ac-49b8-a507-4b452bb944e7\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") "
Oct 31 02:13:44.252578 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.252383    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-podinfo\") pod \"0d39c057-51e1-49f1-85de-facd9e8f15fc\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") "
Oct 31 02:13:44.252578 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.252411    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-credential-socket\") pod \"bfa28245-e7ac-49b8-a507-4b452bb944e7\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") "
Oct 31 02:13:44.252578 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.252529    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa28245-e7ac-49b8-a507-4b452bb944e7-kube-api-access-2wth6\" (UniqueName: \"kubernetes.io/projected/bfa28245-e7ac-49b8-a507-4b452bb944e7-kube-api-access-2wth6\") pod \"bfa28245-e7ac-49b8-a507-4b452bb944e7\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") "
Oct 31 02:13:44.252788 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.252577    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-workload-socket\") pod \"0d39c057-51e1-49f1-85de-facd9e8f15fc\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") "
Oct 31 02:13:44.252788 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.252680    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-workload-certs\") pod \"0d39c057-51e1-49f1-85de-facd9e8f15fc\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") "
Oct 31 02:13:44.252788 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.252779    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-data\") pod \"0d39c057-51e1-49f1-85de-facd9e8f15fc\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") "
Oct 31 02:13:44.252955 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.252828    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-token\" (UniqueName: \"kubernetes.io/projected/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-token\") pod \"bfa28245-e7ac-49b8-a507-4b452bb944e7\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") "
Oct 31 02:13:44.252955 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.252848    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-workload-socket" pod "bfa28245-e7ac-49b8-a507-4b452bb944e7" (UID: "bfa28245-e7ac-49b8-a507-4b452bb944e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:44.252955 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.252864    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-envoy\") pod \"0d39c057-51e1-49f1-85de-facd9e8f15fc\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") "
Oct 31 02:13:44.252955 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.252929    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-podinfo\") pod \"bfa28245-e7ac-49b8-a507-4b452bb944e7\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") "
Oct 31 02:13:44.253179 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.252961    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-data\") pod \"bfa28245-e7ac-49b8-a507-4b452bb944e7\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") "
Oct 31 02:13:44.253179 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.253008    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-envoy\") pod \"bfa28245-e7ac-49b8-a507-4b452bb944e7\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") "
Oct 31 02:13:44.253179 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.253100    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-token\" (UniqueName: \"kubernetes.io/projected/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-token\") pod \"0d39c057-51e1-49f1-85de-facd9e8f15fc\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") "
Oct 31 02:13:44.253357 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.253178    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d39c057-51e1-49f1-85de-facd9e8f15fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d39c057-51e1-49f1-85de-facd9e8f15fc-istiod-ca-cert\") pod \"0d39c057-51e1-49f1-85de-facd9e8f15fc\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") "
Oct 31 02:13:44.253357 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.253240    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-workload-certs" pod "0d39c057-51e1-49f1-85de-facd9e8f15fc" (UID: "0d39c057-51e1-49f1-85de-facd9e8f15fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:44.253357 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.253308    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d39c057-51e1-49f1-85de-facd9e8f15fc-kube-api-access-nm282\" (UniqueName: \"kubernetes.io/projected/0d39c057-51e1-49f1-85de-facd9e8f15fc-kube-api-access-nm282\") pod \"0d39c057-51e1-49f1-85de-facd9e8f15fc\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") "
Oct 31 02:13:44.253357 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.253353    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-workload-certs\") pod \"bfa28245-e7ac-49b8-a507-4b452bb944e7\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") "
Oct 31 02:13:44.253652 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.253384    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-credential-socket\") pod \"0d39c057-51e1-49f1-85de-facd9e8f15fc\" (UID: \"0d39c057-51e1-49f1-85de-facd9e8f15fc\") "
Oct 31 02:13:44.253652 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.253439    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfa28245-e7ac-49b8-a507-4b452bb944e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa28245-e7ac-49b8-a507-4b452bb944e7-istiod-ca-cert\") pod \"bfa28245-e7ac-49b8-a507-4b452bb944e7\" (UID: \"bfa28245-e7ac-49b8-a507-4b452bb944e7\") "
Oct 31 02:13:44.253652 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.253484    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-data" pod "0d39c057-51e1-49f1-85de-facd9e8f15fc" (UID: "0d39c057-51e1-49f1-85de-facd9e8f15fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:44.254516 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.254488    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-workload-socket" pod "0d39c057-51e1-49f1-85de-facd9e8f15fc" (UID: "0d39c057-51e1-49f1-85de-facd9e8f15fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:44.254516 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.254496    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:44.255357 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.254533    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:44.255357 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.254554    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:44.255357 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.254758    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-credential-socket" pod "bfa28245-e7ac-49b8-a507-4b452bb944e7" (UID: "bfa28245-e7ac-49b8-a507-4b452bb944e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:44.255357 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.254787    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfa28245-e7ac-49b8-a507-4b452bb944e7-istiod-ca-cert" pod "bfa28245-e7ac-49b8-a507-4b452bb944e7" (UID: "bfa28245-e7ac-49b8-a507-4b452bb944e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:44.255357 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.255290    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d39c057-51e1-49f1-85de-facd9e8f15fc-istiod-ca-cert" pod "0d39c057-51e1-49f1-85de-facd9e8f15fc" (UID: "0d39c057-51e1-49f1-85de-facd9e8f15fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:44.255748 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.255583    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-data" pod "bfa28245-e7ac-49b8-a507-4b452bb944e7" (UID: "bfa28245-e7ac-49b8-a507-4b452bb944e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:44.256320 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.256048    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-podinfo" pod "0d39c057-51e1-49f1-85de-facd9e8f15fc" (UID: "0d39c057-51e1-49f1-85de-facd9e8f15fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:13:44.257735 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.257707    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-workload-certs" pod "bfa28245-e7ac-49b8-a507-4b452bb944e7" (UID: "bfa28245-e7ac-49b8-a507-4b452bb944e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:44.257943 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.257921    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-credential-socket" pod "0d39c057-51e1-49f1-85de-facd9e8f15fc" (UID: "0d39c057-51e1-49f1-85de-facd9e8f15fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:44.258073 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.258053    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d39c057-51e1-49f1-85de-facd9e8f15fc-kube-api-access-nm282" pod "0d39c057-51e1-49f1-85de-facd9e8f15fc" (UID: "0d39c057-51e1-49f1-85de-facd9e8f15fc"). InnerVolumeSpecName "kube-api-access-nm282". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:44.258225 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.258185    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-envoy" pod "bfa28245-e7ac-49b8-a507-4b452bb944e7" (UID: "bfa28245-e7ac-49b8-a507-4b452bb944e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:44.259015 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.258967    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-podinfo" pod "bfa28245-e7ac-49b8-a507-4b452bb944e7" (UID: "bfa28245-e7ac-49b8-a507-4b452bb944e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:13:44.259276 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.259226    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-envoy" pod "0d39c057-51e1-49f1-85de-facd9e8f15fc" (UID: "0d39c057-51e1-49f1-85de-facd9e8f15fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:44.259362 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.259278    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-token" pod "0d39c057-51e1-49f1-85de-facd9e8f15fc" (UID: "0d39c057-51e1-49f1-85de-facd9e8f15fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:44.261369 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.261333    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-token" pod "bfa28245-e7ac-49b8-a507-4b452bb944e7" (UID: "bfa28245-e7ac-49b8-a507-4b452bb944e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:44.262301 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.262275    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa28245-e7ac-49b8-a507-4b452bb944e7-kube-api-access-2wth6" pod "bfa28245-e7ac-49b8-a507-4b452bb944e7" (UID: "bfa28245-e7ac-49b8-a507-4b452bb944e7"). InnerVolumeSpecName "kube-api-access-2wth6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:44.356818 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.356713    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm282\" (UniqueName: \"kubernetes.io/projected/0d39c057-51e1-49f1-85de-facd9e8f15fc-kube-api-access-nm282\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:44.356818 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.356751    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:44.356818 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.356766    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:44.356818 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.356778    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa28245-e7ac-49b8-a507-4b452bb944e7-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:44.356818 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.356791    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:44.356818 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.356805    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:44.356818 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.356818    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wth6\" (UniqueName: \"kubernetes.io/projected/bfa28245-e7ac-49b8-a507-4b452bb944e7-kube-api-access-2wth6\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:44.357214 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.356832    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:44.357214 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.356845    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:44.357214 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.356856    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:44.357214 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.356869    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:44.357214 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.356882    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:44.357214 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.356893    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa28245-e7ac-49b8-a507-4b452bb944e7-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:44.357214 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.356906    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d39c057-51e1-49f1-85de-facd9e8f15fc-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:44.357214 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.356917    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d39c057-51e1-49f1-85de-facd9e8f15fc-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:44.874211 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.874177    5650 generic.go:356] "Generic (PLEG): container finished" podID="bfa28245-e7ac-49b8-a507-4b452bb944e7" containerID="9a85ecca2d62a9abe6dbf01a8437fbebb8d3643bd1fd44f3b0ddca3adc05e2a4" exitCode=0
Oct 31 02:13:44.874681 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.874228    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"
Oct 31 02:13:44.874681 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.874268    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m" event={"ID":"bfa28245-e7ac-49b8-a507-4b452bb944e7","Type":"ContainerDied","Data":"9a85ecca2d62a9abe6dbf01a8437fbebb8d3643bd1fd44f3b0ddca3adc05e2a4"}
Oct 31 02:13:44.874681 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.874371    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m" event={"ID":"bfa28245-e7ac-49b8-a507-4b452bb944e7","Type":"ContainerDied","Data":"73966d8edb6c33aad4d35e192ee3a42c97bd3cb7429f2e09d43450e622a667bc"}
Oct 31 02:13:44.874681 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.874401    5650 scope.go:122] "RemoveContainer" containerID="9a85ecca2d62a9abe6dbf01a8437fbebb8d3643bd1fd44f3b0ddca3adc05e2a4"
Oct 31 02:13:44.876330 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.876305    5650 generic.go:356] "Generic (PLEG): container finished" podID="0d39c057-51e1-49f1-85de-facd9e8f15fc" containerID="0f3f7de50e68596f8b21df5e920197215208339ecb95644d9fa3cd80501526e0" exitCode=0
Oct 31 02:13:44.876435 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.876371    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"
Oct 31 02:13:44.876435 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.876395    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np" event={"ID":"0d39c057-51e1-49f1-85de-facd9e8f15fc","Type":"ContainerDied","Data":"0f3f7de50e68596f8b21df5e920197215208339ecb95644d9fa3cd80501526e0"}
Oct 31 02:13:44.876546 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.876464    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np" event={"ID":"0d39c057-51e1-49f1-85de-facd9e8f15fc","Type":"ContainerDied","Data":"3ee1c0ed1d608bd838989a4835b814956efdb1e729c494a2361d90f1aefa8693"}
Oct 31 02:13:44.889051 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.889030    5650 scope.go:122] "RemoveContainer" containerID="9a85ecca2d62a9abe6dbf01a8437fbebb8d3643bd1fd44f3b0ddca3adc05e2a4"
Oct 31 02:13:44.889325 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:13:44.889303    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a85ecca2d62a9abe6dbf01a8437fbebb8d3643bd1fd44f3b0ddca3adc05e2a4\": container with ID starting with 9a85ecca2d62a9abe6dbf01a8437fbebb8d3643bd1fd44f3b0ddca3adc05e2a4 not found: ID does not exist" containerID="9a85ecca2d62a9abe6dbf01a8437fbebb8d3643bd1fd44f3b0ddca3adc05e2a4"
Oct 31 02:13:44.889376 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.889334    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a85ecca2d62a9abe6dbf01a8437fbebb8d3643bd1fd44f3b0ddca3adc05e2a4"} err="failed to get container status \"9a85ecca2d62a9abe6dbf01a8437fbebb8d3643bd1fd44f3b0ddca3adc05e2a4\": rpc error: code = NotFound desc = could not find container \"9a85ecca2d62a9abe6dbf01a8437fbebb8d3643bd1fd44f3b0ddca3adc05e2a4\": container with ID starting with 9a85ecca2d62a9abe6dbf01a8437fbebb8d3643bd1fd44f3b0ddca3adc05e2a4 not found: ID does not exist"
Oct 31 02:13:44.889376 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.889352    5650 scope.go:122] "RemoveContainer" containerID="0f3f7de50e68596f8b21df5e920197215208339ecb95644d9fa3cd80501526e0"
Oct 31 02:13:44.901527 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.901501    5650 scope.go:122] "RemoveContainer" containerID="59a60e4b44b4c5a3fb2fafb89a0f33421e9c07de1142c2dd75c40fb56d727ddd"
Oct 31 02:13:44.907413 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.907386    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"]
Oct 31 02:13:44.914123 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.914099    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-ftq5m"]
Oct 31 02:13:44.917513 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.917490    5650 scope.go:122] "RemoveContainer" containerID="0f3f7de50e68596f8b21df5e920197215208339ecb95644d9fa3cd80501526e0"
Oct 31 02:13:44.917910 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:13:44.917883    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f3f7de50e68596f8b21df5e920197215208339ecb95644d9fa3cd80501526e0\": container with ID starting with 0f3f7de50e68596f8b21df5e920197215208339ecb95644d9fa3cd80501526e0 not found: ID does not exist" containerID="0f3f7de50e68596f8b21df5e920197215208339ecb95644d9fa3cd80501526e0"
Oct 31 02:13:44.917986 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.917919    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f3f7de50e68596f8b21df5e920197215208339ecb95644d9fa3cd80501526e0"} err="failed to get container status \"0f3f7de50e68596f8b21df5e920197215208339ecb95644d9fa3cd80501526e0\": rpc error: code = NotFound desc = could not find container \"0f3f7de50e68596f8b21df5e920197215208339ecb95644d9fa3cd80501526e0\": container with ID starting with 0f3f7de50e68596f8b21df5e920197215208339ecb95644d9fa3cd80501526e0 not found: ID does not exist"
Oct 31 02:13:44.917986 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.917937    5650 scope.go:122] "RemoveContainer" containerID="59a60e4b44b4c5a3fb2fafb89a0f33421e9c07de1142c2dd75c40fb56d727ddd"
Oct 31 02:13:44.918232 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:13:44.918213    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59a60e4b44b4c5a3fb2fafb89a0f33421e9c07de1142c2dd75c40fb56d727ddd\": container with ID starting with 59a60e4b44b4c5a3fb2fafb89a0f33421e9c07de1142c2dd75c40fb56d727ddd not found: ID does not exist" containerID="59a60e4b44b4c5a3fb2fafb89a0f33421e9c07de1142c2dd75c40fb56d727ddd"
Oct 31 02:13:44.918287 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.918238    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59a60e4b44b4c5a3fb2fafb89a0f33421e9c07de1142c2dd75c40fb56d727ddd"} err="failed to get container status \"59a60e4b44b4c5a3fb2fafb89a0f33421e9c07de1142c2dd75c40fb56d727ddd\": rpc error: code = NotFound desc = could not find container \"59a60e4b44b4c5a3fb2fafb89a0f33421e9c07de1142c2dd75c40fb56d727ddd\": container with ID starting with 59a60e4b44b4c5a3fb2fafb89a0f33421e9c07de1142c2dd75c40fb56d727ddd not found: ID does not exist"
Oct 31 02:13:44.929789 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.929758    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"]
Oct 31 02:13:44.937968 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:44.937940    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-6c54a1-openshift-default-6944886cdf-rx2np"]
Oct 31 02:13:45.462267 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:45.462221    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"]
Oct 31 02:13:45.463134 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:45.463083    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72" podUID="e09af575-0369-4421-9326-9836c5910c67" containerName="istio-proxy" containerID="cri-o://4260e90a47600f3ef95eb71d4cbaae8cc0efc25a68787cfc41b3b33a766efd93" gracePeriod=30
Oct 31 02:13:46.122127 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.122096    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d39c057-51e1-49f1-85de-facd9e8f15fc" path="/var/lib/kubelet/pods/0d39c057-51e1-49f1-85de-facd9e8f15fc/volumes"
Oct 31 02:13:46.123602 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.123576    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfa28245-e7ac-49b8-a507-4b452bb944e7" path="/var/lib/kubelet/pods/bfa28245-e7ac-49b8-a507-4b452bb944e7/volumes"
Oct 31 02:13:46.431130 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.431105    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:13:46.529568 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.529535    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-envoy\") pod \"e9c1e7ce-8191-44fd-a838-b9718752488a\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") "
Oct 31 02:13:46.529769 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.529592    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-workload-certs\") pod \"e9c1e7ce-8191-44fd-a838-b9718752488a\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") "
Oct 31 02:13:46.529769 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.529656    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-data\") pod \"e9c1e7ce-8191-44fd-a838-b9718752488a\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") "
Oct 31 02:13:46.529769 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.529678    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c1e7ce-8191-44fd-a838-b9718752488a-kube-api-access-7pgt2\" (UniqueName: \"kubernetes.io/projected/e9c1e7ce-8191-44fd-a838-b9718752488a-kube-api-access-7pgt2\") pod \"e9c1e7ce-8191-44fd-a838-b9718752488a\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") "
Oct 31 02:13:46.529955 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.529807    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-workload-socket\") pod \"e9c1e7ce-8191-44fd-a838-b9718752488a\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") "
Oct 31 02:13:46.529955 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.529840    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-credential-socket\") pod \"e9c1e7ce-8191-44fd-a838-b9718752488a\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") "
Oct 31 02:13:46.529955 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.529940    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c1e7ce-8191-44fd-a838-b9718752488a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c1e7ce-8191-44fd-a838-b9718752488a-istiod-ca-cert\") pod \"e9c1e7ce-8191-44fd-a838-b9718752488a\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") "
Oct 31 02:13:46.530784 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.529947    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-workload-certs" pod "e9c1e7ce-8191-44fd-a838-b9718752488a" (UID: "e9c1e7ce-8191-44fd-a838-b9718752488a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:46.530784 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.529998    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-token\" (UniqueName: \"kubernetes.io/projected/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-token\") pod \"e9c1e7ce-8191-44fd-a838-b9718752488a\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") "
Oct 31 02:13:46.530784 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.530184    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-workload-socket" pod "e9c1e7ce-8191-44fd-a838-b9718752488a" (UID: "e9c1e7ce-8191-44fd-a838-b9718752488a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:46.530784 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.530336    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-credential-socket" pod "e9c1e7ce-8191-44fd-a838-b9718752488a" (UID: "e9c1e7ce-8191-44fd-a838-b9718752488a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:46.530784 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.530463    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c1e7ce-8191-44fd-a838-b9718752488a-istiod-ca-cert" pod "e9c1e7ce-8191-44fd-a838-b9718752488a" (UID: "e9c1e7ce-8191-44fd-a838-b9718752488a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:46.530784 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.530555    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-data" pod "e9c1e7ce-8191-44fd-a838-b9718752488a" (UID: "e9c1e7ce-8191-44fd-a838-b9718752488a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:46.530784 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.530206    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-podinfo\") pod \"e9c1e7ce-8191-44fd-a838-b9718752488a\" (UID: \"e9c1e7ce-8191-44fd-a838-b9718752488a\") "
Oct 31 02:13:46.531769 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.531743    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:46.531872 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.531774    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:46.531872 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.531788    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:46.531872 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.531805    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:46.531872 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.531817    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c1e7ce-8191-44fd-a838-b9718752488a-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:46.533018 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.532990    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-podinfo" pod "e9c1e7ce-8191-44fd-a838-b9718752488a" (UID: "e9c1e7ce-8191-44fd-a838-b9718752488a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:13:46.533142 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.533021    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-envoy" pod "e9c1e7ce-8191-44fd-a838-b9718752488a" (UID: "e9c1e7ce-8191-44fd-a838-b9718752488a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:46.533142 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.533054    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c1e7ce-8191-44fd-a838-b9718752488a-kube-api-access-7pgt2" pod "e9c1e7ce-8191-44fd-a838-b9718752488a" (UID: "e9c1e7ce-8191-44fd-a838-b9718752488a"). InnerVolumeSpecName "kube-api-access-7pgt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:46.533289 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.533280    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-token" pod "e9c1e7ce-8191-44fd-a838-b9718752488a" (UID: "e9c1e7ce-8191-44fd-a838-b9718752488a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:46.633747 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.633664    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:46.633909 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.633816    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:46.633909 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.633836    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pgt2\" (UniqueName: \"kubernetes.io/projected/e9c1e7ce-8191-44fd-a838-b9718752488a-kube-api-access-7pgt2\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:46.633909 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.633846    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c1e7ce-8191-44fd-a838-b9718752488a-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:46.900648 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.900544    5650 generic.go:356] "Generic (PLEG): container finished" podID="e9c1e7ce-8191-44fd-a838-b9718752488a" containerID="2f5e6715f052080d3c5c0d7449e23e6ff65587e0ce755b1785ad3e9ca0e0f339" exitCode=0
Oct 31 02:13:46.900648 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.900634    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"
Oct 31 02:13:46.900857 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.900649    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr" event={"ID":"e9c1e7ce-8191-44fd-a838-b9718752488a","Type":"ContainerDied","Data":"2f5e6715f052080d3c5c0d7449e23e6ff65587e0ce755b1785ad3e9ca0e0f339"}
Oct 31 02:13:46.900857 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.900748    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr" event={"ID":"e9c1e7ce-8191-44fd-a838-b9718752488a","Type":"ContainerDied","Data":"92eab140fa1cfd50f3c2bf7de2f5395ccffa46f4ef49049d53e02638bbc8a7da"}
Oct 31 02:13:46.900857 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.900777    5650 scope.go:122] "RemoveContainer" containerID="2f5e6715f052080d3c5c0d7449e23e6ff65587e0ce755b1785ad3e9ca0e0f339"
Oct 31 02:13:46.915561 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.915535    5650 scope.go:122] "RemoveContainer" containerID="2f5e6715f052080d3c5c0d7449e23e6ff65587e0ce755b1785ad3e9ca0e0f339"
Oct 31 02:13:46.915961 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:13:46.915934    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f5e6715f052080d3c5c0d7449e23e6ff65587e0ce755b1785ad3e9ca0e0f339\": container with ID starting with 2f5e6715f052080d3c5c0d7449e23e6ff65587e0ce755b1785ad3e9ca0e0f339 not found: ID does not exist" containerID="2f5e6715f052080d3c5c0d7449e23e6ff65587e0ce755b1785ad3e9ca0e0f339"
Oct 31 02:13:46.916073 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.915971    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f5e6715f052080d3c5c0d7449e23e6ff65587e0ce755b1785ad3e9ca0e0f339"} err="failed to get container status \"2f5e6715f052080d3c5c0d7449e23e6ff65587e0ce755b1785ad3e9ca0e0f339\": rpc error: code = NotFound desc = could not find container \"2f5e6715f052080d3c5c0d7449e23e6ff65587e0ce755b1785ad3e9ca0e0f339\": container with ID starting with 2f5e6715f052080d3c5c0d7449e23e6ff65587e0ce755b1785ad3e9ca0e0f339 not found: ID does not exist"
Oct 31 02:13:46.966119 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.966075    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"]
Oct 31 02:13:46.980391 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:46.980346    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-qbtsr"]
Oct 31 02:13:48.122152 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:48.122116    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c1e7ce-8191-44fd-a838-b9718752488a" path="/var/lib/kubelet/pods/e9c1e7ce-8191-44fd-a838-b9718752488a/volumes"
Oct 31 02:13:48.232807 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:48.232772    5650 scope.go:122] "RemoveContainer" containerID="d00107c2f550afb50012b71ef3b7f3403eec9df505d2b493e3c1ac3df39626f5"
Oct 31 02:13:48.252550 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:48.252515    5650 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:13:48.252689 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:48.252581    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2" podUID="214c7868-1e53-463f-8247-bed23603debd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:13:49.340914 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.340884    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p"]
Oct 31 02:13:49.386587 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.386531    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw"]
Oct 31 02:13:49.684311 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.684281    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 02:13:49.802934 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.802876    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-credential-socket\") pod \"214c7868-1e53-463f-8247-bed23603debd\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") "
Oct 31 02:13:49.802934 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.802949    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/214c7868-1e53-463f-8247-bed23603debd-istio-token\" (UniqueName: \"kubernetes.io/projected/214c7868-1e53-463f-8247-bed23603debd-istio-token\") pod \"214c7868-1e53-463f-8247-bed23603debd\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") "
Oct 31 02:13:49.803631 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.803165    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/214c7868-1e53-463f-8247-bed23603debd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/214c7868-1e53-463f-8247-bed23603debd-istio-podinfo\") pod \"214c7868-1e53-463f-8247-bed23603debd\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") "
Oct 31 02:13:49.803631 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.803269    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/214c7868-1e53-463f-8247-bed23603debd-kube-api-access-xg9p6\" (UniqueName: \"kubernetes.io/projected/214c7868-1e53-463f-8247-bed23603debd-kube-api-access-xg9p6\") pod \"214c7868-1e53-463f-8247-bed23603debd\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") "
Oct 31 02:13:49.803631 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.803295    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/214c7868-1e53-463f-8247-bed23603debd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/214c7868-1e53-463f-8247-bed23603debd-istiod-ca-cert\") pod \"214c7868-1e53-463f-8247-bed23603debd\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") "
Oct 31 02:13:49.803631 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.803321    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-istio-envoy\") pod \"214c7868-1e53-463f-8247-bed23603debd\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") "
Oct 31 02:13:49.803631 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.803365    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-workload-socket\") pod \"214c7868-1e53-463f-8247-bed23603debd\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") "
Oct 31 02:13:49.803631 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.803386    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-istio-data\") pod \"214c7868-1e53-463f-8247-bed23603debd\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") "
Oct 31 02:13:49.803631 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.803466    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-workload-certs\") pod \"214c7868-1e53-463f-8247-bed23603debd\" (UID: \"214c7868-1e53-463f-8247-bed23603debd\") "
Oct 31 02:13:49.803903 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.803655    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-credential-socket" pod "214c7868-1e53-463f-8247-bed23603debd" (UID: "214c7868-1e53-463f-8247-bed23603debd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:49.804133 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.804113    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:49.804563 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.804507    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-workload-socket" pod "214c7868-1e53-463f-8247-bed23603debd" (UID: "214c7868-1e53-463f-8247-bed23603debd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:49.804705 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.804581    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-istio-data" pod "214c7868-1e53-463f-8247-bed23603debd" (UID: "214c7868-1e53-463f-8247-bed23603debd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:49.804705 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.804597    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-workload-certs" pod "214c7868-1e53-463f-8247-bed23603debd" (UID: "214c7868-1e53-463f-8247-bed23603debd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:49.806975 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.806949    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/214c7868-1e53-463f-8247-bed23603debd-istio-token" pod "214c7868-1e53-463f-8247-bed23603debd" (UID: "214c7868-1e53-463f-8247-bed23603debd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:49.807800 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.807771    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/214c7868-1e53-463f-8247-bed23603debd-istio-podinfo" pod "214c7868-1e53-463f-8247-bed23603debd" (UID: "214c7868-1e53-463f-8247-bed23603debd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:13:49.808422 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.808398    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-istio-envoy" pod "214c7868-1e53-463f-8247-bed23603debd" (UID: "214c7868-1e53-463f-8247-bed23603debd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:49.808538 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.808469    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/214c7868-1e53-463f-8247-bed23603debd-kube-api-access-xg9p6" pod "214c7868-1e53-463f-8247-bed23603debd" (UID: "214c7868-1e53-463f-8247-bed23603debd"). InnerVolumeSpecName "kube-api-access-xg9p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:49.808837 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.808816    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/214c7868-1e53-463f-8247-bed23603debd-istiod-ca-cert" pod "214c7868-1e53-463f-8247-bed23603debd" (UID: "214c7868-1e53-463f-8247-bed23603debd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:49.932550 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.926567    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/214c7868-1e53-463f-8247-bed23603debd-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:49.932550 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.926606    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/214c7868-1e53-463f-8247-bed23603debd-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:49.932550 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.926623    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg9p6\" (UniqueName: \"kubernetes.io/projected/214c7868-1e53-463f-8247-bed23603debd-kube-api-access-xg9p6\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:49.932550 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.926642    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/214c7868-1e53-463f-8247-bed23603debd-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:49.932550 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.926657    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:49.932550 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.926672    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:49.932550 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.926687    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:49.932550 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:49.926701    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/214c7868-1e53-463f-8247-bed23603debd-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:50.202215 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.202184    5650 generic.go:356] "Generic (PLEG): container finished" podID="214c7868-1e53-463f-8247-bed23603debd" containerID="a73b3a3e25c8fc6a3df26a76791f6ccf7ad5e2ed362aaefb3417df8b4f789cb2" exitCode=0
Oct 31 02:13:50.202415 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.202400    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"
Oct 31 02:13:50.219915 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.216832    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2" event={"ID":"214c7868-1e53-463f-8247-bed23603debd","Type":"ContainerDied","Data":"a73b3a3e25c8fc6a3df26a76791f6ccf7ad5e2ed362aaefb3417df8b4f789cb2"}
Oct 31 02:13:50.219915 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.216889    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2" event={"ID":"214c7868-1e53-463f-8247-bed23603debd","Type":"ContainerDied","Data":"662f85f1b47d9491fbd41c3f49e8c6d7ccc741ca916e14ccce1f4715acc474ba"}
Oct 31 02:13:50.219915 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.216912    5650 scope.go:122] "RemoveContainer" containerID="a73b3a3e25c8fc6a3df26a76791f6ccf7ad5e2ed362aaefb3417df8b4f789cb2"
Oct 31 02:13:50.240931 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.240904    5650 scope.go:122] "RemoveContainer" containerID="a73b3a3e25c8fc6a3df26a76791f6ccf7ad5e2ed362aaefb3417df8b4f789cb2"
Oct 31 02:13:50.241339 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:13:50.241315    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a73b3a3e25c8fc6a3df26a76791f6ccf7ad5e2ed362aaefb3417df8b4f789cb2\": container with ID starting with a73b3a3e25c8fc6a3df26a76791f6ccf7ad5e2ed362aaefb3417df8b4f789cb2 not found: ID does not exist" containerID="a73b3a3e25c8fc6a3df26a76791f6ccf7ad5e2ed362aaefb3417df8b4f789cb2"
Oct 31 02:13:50.241499 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.241477    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73b3a3e25c8fc6a3df26a76791f6ccf7ad5e2ed362aaefb3417df8b4f789cb2"} err="failed to get container status \"a73b3a3e25c8fc6a3df26a76791f6ccf7ad5e2ed362aaefb3417df8b4f789cb2\": rpc error: code = NotFound desc = could not find container \"a73b3a3e25c8fc6a3df26a76791f6ccf7ad5e2ed362aaefb3417df8b4f789cb2\": container with ID starting with a73b3a3e25c8fc6a3df26a76791f6ccf7ad5e2ed362aaefb3417df8b4f789cb2 not found: ID does not exist"
Oct 31 02:13:50.248405 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.248367    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"]
Oct 31 02:13:50.255925 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.255897    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-5vgt2"]
Oct 31 02:13:50.860804 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.860777    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 02:13:50.997160 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.997119    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e09af575-0369-4421-9326-9836c5910c67-kube-api-access-slx9z\" (UniqueName: \"kubernetes.io/projected/e09af575-0369-4421-9326-9836c5910c67-kube-api-access-slx9z\") pod \"e09af575-0369-4421-9326-9836c5910c67\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") "
Oct 31 02:13:50.997363 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.997181    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-credential-socket\") pod \"e09af575-0369-4421-9326-9836c5910c67\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") "
Oct 31 02:13:50.997363 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.997230    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e09af575-0369-4421-9326-9836c5910c67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09af575-0369-4421-9326-9836c5910c67-istiod-ca-cert\") pod \"e09af575-0369-4421-9326-9836c5910c67\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") "
Oct 31 02:13:50.997513 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.997368    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-istio-envoy\") pod \"e09af575-0369-4421-9326-9836c5910c67\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") "
Oct 31 02:13:50.997513 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.997406    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-workload-socket\") pod \"e09af575-0369-4421-9326-9836c5910c67\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") "
Oct 31 02:13:50.997627 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.997565    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e09af575-0369-4421-9326-9836c5910c67-istio-token\" (UniqueName: \"kubernetes.io/projected/e09af575-0369-4421-9326-9836c5910c67-istio-token\") pod \"e09af575-0369-4421-9326-9836c5910c67\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") "
Oct 31 02:13:50.997684 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.997675    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-istio-data\") pod \"e09af575-0369-4421-9326-9836c5910c67\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") "
Oct 31 02:13:50.997819 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.997795    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-workload-certs\") pod \"e09af575-0369-4421-9326-9836c5910c67\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") "
Oct 31 02:13:50.997953 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.997874    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e09af575-0369-4421-9326-9836c5910c67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09af575-0369-4421-9326-9836c5910c67-istio-podinfo\") pod \"e09af575-0369-4421-9326-9836c5910c67\" (UID: \"e09af575-0369-4421-9326-9836c5910c67\") "
Oct 31 02:13:50.998192 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.998129    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-credential-socket" pod "e09af575-0369-4421-9326-9836c5910c67" (UID: "e09af575-0369-4421-9326-9836c5910c67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:50.998512 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.998345    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-workload-socket" pod "e09af575-0369-4421-9326-9836c5910c67" (UID: "e09af575-0369-4421-9326-9836c5910c67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:50.998512 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.998396    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e09af575-0369-4421-9326-9836c5910c67-istiod-ca-cert" pod "e09af575-0369-4421-9326-9836c5910c67" (UID: "e09af575-0369-4421-9326-9836c5910c67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:13:50.998880 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.998861    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:50.999749 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.998881    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:50.999749 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.998892    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09af575-0369-4421-9326-9836c5910c67-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:50.999883 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.999776    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09af575-0369-4421-9326-9836c5910c67-kube-api-access-slx9z" pod "e09af575-0369-4421-9326-9836c5910c67" (UID: "e09af575-0369-4421-9326-9836c5910c67"). InnerVolumeSpecName "kube-api-access-slx9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:50.999941 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:50.999905    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-istio-envoy" pod "e09af575-0369-4421-9326-9836c5910c67" (UID: "e09af575-0369-4421-9326-9836c5910c67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:51.001292 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:51.001266    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-istio-data" pod "e09af575-0369-4421-9326-9836c5910c67" (UID: "e09af575-0369-4421-9326-9836c5910c67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:51.001403 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:51.001309    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-workload-certs" pod "e09af575-0369-4421-9326-9836c5910c67" (UID: "e09af575-0369-4421-9326-9836c5910c67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:51.002431 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:51.002405    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09af575-0369-4421-9326-9836c5910c67-istio-token" pod "e09af575-0369-4421-9326-9836c5910c67" (UID: "e09af575-0369-4421-9326-9836c5910c67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:51.010570 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:51.010543    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e09af575-0369-4421-9326-9836c5910c67-istio-podinfo" pod "e09af575-0369-4421-9326-9836c5910c67" (UID: "e09af575-0369-4421-9326-9836c5910c67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:13:51.103064 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:51.102968    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:51.103480 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:51.103261    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09af575-0369-4421-9326-9836c5910c67-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:51.103480 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:51.103285    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slx9z\" (UniqueName: \"kubernetes.io/projected/e09af575-0369-4421-9326-9836c5910c67-kube-api-access-slx9z\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:51.103480 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:51.103299    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:51.103480 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:51.103313    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e09af575-0369-4421-9326-9836c5910c67-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:51.103480 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:51.103326    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09af575-0369-4421-9326-9836c5910c67-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:51.214740 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:51.214693    5650 generic.go:356] "Generic (PLEG): container finished" podID="e09af575-0369-4421-9326-9836c5910c67" containerID="4260e90a47600f3ef95eb71d4cbaae8cc0efc25a68787cfc41b3b33a766efd93" exitCode=0
Oct 31 02:13:51.214740 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:51.214741    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"
Oct 31 02:13:51.215013 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:51.214775    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72" event={"ID":"e09af575-0369-4421-9326-9836c5910c67","Type":"ContainerDied","Data":"4260e90a47600f3ef95eb71d4cbaae8cc0efc25a68787cfc41b3b33a766efd93"}
Oct 31 02:13:51.215013 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:51.214830    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72" event={"ID":"e09af575-0369-4421-9326-9836c5910c67","Type":"ContainerDied","Data":"52cd8667ba64ecdccc59c985906f3c60d531019bb908658275a08a73c278a576"}
Oct 31 02:13:51.215013 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:51.214861    5650 scope.go:122] "RemoveContainer" containerID="4260e90a47600f3ef95eb71d4cbaae8cc0efc25a68787cfc41b3b33a766efd93"
Oct 31 02:13:51.229512 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:51.229490    5650 scope.go:122] "RemoveContainer" containerID="4260e90a47600f3ef95eb71d4cbaae8cc0efc25a68787cfc41b3b33a766efd93"
Oct 31 02:13:51.229806 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:13:51.229788    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4260e90a47600f3ef95eb71d4cbaae8cc0efc25a68787cfc41b3b33a766efd93\": container with ID starting with 4260e90a47600f3ef95eb71d4cbaae8cc0efc25a68787cfc41b3b33a766efd93 not found: ID does not exist" containerID="4260e90a47600f3ef95eb71d4cbaae8cc0efc25a68787cfc41b3b33a766efd93"
Oct 31 02:13:51.229872 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:51.229817    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4260e90a47600f3ef95eb71d4cbaae8cc0efc25a68787cfc41b3b33a766efd93"} err="failed to get container status \"4260e90a47600f3ef95eb71d4cbaae8cc0efc25a68787cfc41b3b33a766efd93\": rpc error: code = NotFound desc = could not find container \"4260e90a47600f3ef95eb71d4cbaae8cc0efc25a68787cfc41b3b33a766efd93\": container with ID starting with 4260e90a47600f3ef95eb71d4cbaae8cc0efc25a68787cfc41b3b33a766efd93 not found: ID does not exist"
Oct 31 02:13:51.252359 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:51.252316    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"]
Oct 31 02:13:51.259247 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:51.259198    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-cpc72"]
Oct 31 02:13:52.122003 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:52.121968    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="214c7868-1e53-463f-8247-bed23603debd" path="/var/lib/kubelet/pods/214c7868-1e53-463f-8247-bed23603debd/volumes"
Oct 31 02:13:52.122626 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:52.122609    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e09af575-0369-4421-9326-9836c5910c67" path="/var/lib/kubelet/pods/e09af575-0369-4421-9326-9836c5910c67/volumes"
Oct 31 02:13:54.342379 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:54.342332    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p" podUID="7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e" containerName="payload-processing" containerID="cri-o://b07ae836aa64d66b7686b5255f24807f93e13ddb1686c70b9f39d6e698af38ad" gracePeriod=25
Oct 31 02:13:54.388765 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:54.388528    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw" podUID="35f0ddb1-3758-4a14-a2e2-2c7a928b3564" containerName="payload-pre-processing" containerID="cri-o://97d4a78fcd99709176951137a6a5e07e325bafa066bb7663b8021cd346be6e69" gracePeriod=25
Oct 31 02:13:56.969567 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:56.969524    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p" podUID="7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:13:56.969976 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:56.969657    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw" podUID="35f0ddb1-3758-4a14-a2e2-2c7a928b3564" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:13:58.830476 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:13:58.829782    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:13:58.830476 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:58.829880    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" containerID="cri-o://609e298c7f01f234faefe0ed000a1ec0a10ff6f34e274fd54c4abc5f63fb36d0" gracePeriod=45
Oct 31 02:13:59.058502 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:59.058461    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": dial tcp 10.134.0.83:8000: connect: connection refused"
Oct 31 02:13:59.312334 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:59.312303    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" event={"ID":"a2feec49-d795-4179-8e7b-cbb631a9b5e8","Type":"ContainerDied","Data":"609e298c7f01f234faefe0ed000a1ec0a10ff6f34e274fd54c4abc5f63fb36d0"}
Oct 31 02:13:59.312616 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:59.312255    5650 generic.go:356] "Generic (PLEG): container finished" podID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerID="609e298c7f01f234faefe0ed000a1ec0a10ff6f34e274fd54c4abc5f63fb36d0" exitCode=0
Oct 31 02:13:59.403015 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:59.402984    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 02:13:59.426870 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:59.426830    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2feec49-d795-4179-8e7b-cbb631a9b5e8-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2feec49-d795-4179-8e7b-cbb631a9b5e8-tls-certs\") pod \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\" (UID: \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\") "
Oct 31 02:13:59.427159 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:59.427095    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-home\" (UniqueName: \"kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-home\") pod \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\" (UID: \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\") "
Oct 31 02:13:59.427325 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:59.427194    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-model-cache\") pod \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\" (UID: \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\") "
Oct 31 02:13:59.427325 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:59.427264    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-dshm\") pod \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\" (UID: \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\") "
Oct 31 02:13:59.427544 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:59.427343    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2feec49-d795-4179-8e7b-cbb631a9b5e8-kube-api-access-7mddc\" (UniqueName: \"kubernetes.io/projected/a2feec49-d795-4179-8e7b-cbb631a9b5e8-kube-api-access-7mddc\") pod \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\" (UID: \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\") "
Oct 31 02:13:59.427544 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:59.427466    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-tmp-dir\") pod \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\" (UID: \"a2feec49-d795-4179-8e7b-cbb631a9b5e8\") "
Oct 31 02:13:59.429058 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:59.429026    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-tmp-dir" pod "a2feec49-d795-4179-8e7b-cbb631a9b5e8" (UID: "a2feec49-d795-4179-8e7b-cbb631a9b5e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:59.430543 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:59.430513    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2feec49-d795-4179-8e7b-cbb631a9b5e8-tls-certs" pod "a2feec49-d795-4179-8e7b-cbb631a9b5e8" (UID: "a2feec49-d795-4179-8e7b-cbb631a9b5e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:13:59.430646 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:59.430562    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-model-cache" pod "a2feec49-d795-4179-8e7b-cbb631a9b5e8" (UID: "a2feec49-d795-4179-8e7b-cbb631a9b5e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:59.431344 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:59.430984    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-home" pod "a2feec49-d795-4179-8e7b-cbb631a9b5e8" (UID: "a2feec49-d795-4179-8e7b-cbb631a9b5e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:59.432913 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:59.432886    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2feec49-d795-4179-8e7b-cbb631a9b5e8-kube-api-access-7mddc" pod "a2feec49-d795-4179-8e7b-cbb631a9b5e8" (UID: "a2feec49-d795-4179-8e7b-cbb631a9b5e8"). InnerVolumeSpecName "kube-api-access-7mddc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:13:59.433729 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:59.433700    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-dshm" pod "a2feec49-d795-4179-8e7b-cbb631a9b5e8" (UID: "a2feec49-d795-4179-8e7b-cbb631a9b5e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:13:59.530358 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:59.530277    5650 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-home\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:59.530358 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:59.530305    5650 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-model-cache\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:59.530358 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:59.530314    5650 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-dshm\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:59.530358 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:59.530326    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mddc\" (UniqueName: \"kubernetes.io/projected/a2feec49-d795-4179-8e7b-cbb631a9b5e8-kube-api-access-7mddc\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:59.530358 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:59.530335    5650 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2feec49-d795-4179-8e7b-cbb631a9b5e8-tmp-dir\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:13:59.530358 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:13:59.530346    5650 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2feec49-d795-4179-8e7b-cbb631a9b5e8-tls-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:00.324175 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:00.324141    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"
Oct 31 02:14:00.324652 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:00.324174    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8" event={"ID":"a2feec49-d795-4179-8e7b-cbb631a9b5e8","Type":"ContainerDied","Data":"a0e371d133f81805f5aeb71b91a9ac8dec8940da9ce172af1e0afbc128647747"}
Oct 31 02:14:00.324652 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:00.324232    5650 scope.go:122] "RemoveContainer" containerID="609e298c7f01f234faefe0ed000a1ec0a10ff6f34e274fd54c4abc5f63fb36d0"
Oct 31 02:14:00.392513 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:00.392357    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"]
Oct 31 02:14:00.404319 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:00.404249    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-be6935/distinct-2-w4-be6935-kserve-649ddf5cdb-wm6h8"]
Oct 31 02:14:02.122079 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:02.122044    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" path="/var/lib/kubelet/pods/a2feec49-d795-4179-8e7b-cbb631a9b5e8/volumes"
Oct 31 02:14:06.036521 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.036481    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr"]
Oct 31 02:14:06.037353 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037331    5650 cpu_manager.go:414] "containerMap: removing container" podUID="0d39c057-51e1-49f1-85de-facd9e8f15fc" containerName="istio-proxy"
Oct 31 02:14:06.037353 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037350    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d39c057-51e1-49f1-85de-facd9e8f15fc" containerName="istio-proxy"
Oct 31 02:14:06.037353 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037363    5650 cpu_manager.go:414] "containerMap: removing container" podUID="e9c1e7ce-8191-44fd-a838-b9718752488a" containerName="istio-proxy"
Oct 31 02:14:06.037353 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037369    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c1e7ce-8191-44fd-a838-b9718752488a" containerName="istio-proxy"
Oct 31 02:14:06.037654 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037382    5650 cpu_manager.go:414] "containerMap: removing container" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main"
Oct 31 02:14:06.037654 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037387    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main"
Oct 31 02:14:06.037654 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037396    5650 cpu_manager.go:414] "containerMap: removing container" podUID="bfa28245-e7ac-49b8-a507-4b452bb944e7" containerName="istio-proxy"
Oct 31 02:14:06.037654 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037401    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa28245-e7ac-49b8-a507-4b452bb944e7" containerName="istio-proxy"
Oct 31 02:14:06.037654 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037413    5650 cpu_manager.go:414] "containerMap: removing container" podUID="e09af575-0369-4421-9326-9836c5910c67" containerName="istio-proxy"
Oct 31 02:14:06.037654 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037419    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e09af575-0369-4421-9326-9836c5910c67" containerName="istio-proxy"
Oct 31 02:14:06.037654 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037430    5650 cpu_manager.go:414] "containerMap: removing container" podUID="214c7868-1e53-463f-8247-bed23603debd" containerName="istio-proxy"
Oct 31 02:14:06.037654 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037435    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="214c7868-1e53-463f-8247-bed23603debd" containerName="istio-proxy"
Oct 31 02:14:06.037654 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037441    5650 cpu_manager.go:414] "containerMap: removing container" podUID="2cb82515-c5d2-47a6-af96-d0da5a75bcd3" containerName="istio-proxy"
Oct 31 02:14:06.037654 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037461    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb82515-c5d2-47a6-af96-d0da5a75bcd3" containerName="istio-proxy"
Oct 31 02:14:06.037654 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037507    5650 cpu_manager.go:414] "containerMap: removing container" podUID="0d39c057-51e1-49f1-85de-facd9e8f15fc" containerName="istio-proxy"
Oct 31 02:14:06.037654 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037514    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d39c057-51e1-49f1-85de-facd9e8f15fc" containerName="istio-proxy"
Oct 31 02:14:06.037654 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037553    5650 cpu_manager.go:414] "containerMap: removing container" podUID="014b9592-996c-4034-8453-04e92db14801" containerName="revoke-keys"
Oct 31 02:14:06.037654 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037560    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014b9592-996c-4034-8453-04e92db14801" containerName="revoke-keys"
Oct 31 02:14:06.037654 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037571    5650 cpu_manager.go:414] "containerMap: removing container" podUID="247b6d6e-faaf-44e3-bf73-73fbb620f5e0" containerName="istio-proxy"
Oct 31 02:14:06.037654 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037576    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="247b6d6e-faaf-44e3-bf73-73fbb620f5e0" containerName="istio-proxy"
Oct 31 02:14:06.037654 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037584    5650 cpu_manager.go:414] "containerMap: removing container" podUID="247b6d6e-faaf-44e3-bf73-73fbb620f5e0" containerName="istio-proxy"
Oct 31 02:14:06.037654 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037588    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="247b6d6e-faaf-44e3-bf73-73fbb620f5e0" containerName="istio-proxy"
Oct 31 02:14:06.038202 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037731    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d39c057-51e1-49f1-85de-facd9e8f15fc" containerName="istio-proxy"
Oct 31 02:14:06.038202 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037738    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c1e7ce-8191-44fd-a838-b9718752488a" containerName="istio-proxy"
Oct 31 02:14:06.038202 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037746    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2feec49-d795-4179-8e7b-cbb631a9b5e8" containerName="main"
Oct 31 02:14:06.038202 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037752    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa28245-e7ac-49b8-a507-4b452bb944e7" containerName="istio-proxy"
Oct 31 02:14:06.038202 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037758    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="e09af575-0369-4421-9326-9836c5910c67" containerName="istio-proxy"
Oct 31 02:14:06.038202 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037764    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="214c7868-1e53-463f-8247-bed23603debd" containerName="istio-proxy"
Oct 31 02:14:06.038202 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037769    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb82515-c5d2-47a6-af96-d0da5a75bcd3" containerName="istio-proxy"
Oct 31 02:14:06.038202 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037779    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe7e7396-a227-4486-97ba-8b48945cc1d3" containerName="istio-proxy"
Oct 31 02:14:06.038202 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037792    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="014b9592-996c-4034-8453-04e92db14801" containerName="revoke-keys"
Oct 31 02:14:06.038202 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037798    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="247b6d6e-faaf-44e3-bf73-73fbb620f5e0" containerName="istio-proxy"
Oct 31 02:14:06.038202 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.037805    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="247b6d6e-faaf-44e3-bf73-73fbb620f5e0" containerName="istio-proxy"
Oct 31 02:14:06.043584 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.043558    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr"
Oct 31 02:14:06.048505 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.048478    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:14:06.049726 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.049696    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr"]
Oct 31 02:14:06.050512 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.050441    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:14:06.238259 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.238211    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f58bf2-680c-4c93-81fa-2003efe55968-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr\" (UID: \"31f58bf2-680c-4c93-81fa-2003efe55968\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr"
Oct 31 02:14:06.238490 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.238437    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6mg8\" (UniqueName: \"kubernetes.io/projected/31f58bf2-680c-4c93-81fa-2003efe55968-kube-api-access-v6mg8\") pod \"maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr\" (UID: \"31f58bf2-680c-4c93-81fa-2003efe55968\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr"
Oct 31 02:14:06.238657 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.238630    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f58bf2-680c-4c93-81fa-2003efe55968-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr\" (UID: \"31f58bf2-680c-4c93-81fa-2003efe55968\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr"
Oct 31 02:14:06.340091 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.339995    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f58bf2-680c-4c93-81fa-2003efe55968-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr\" (UID: \"31f58bf2-680c-4c93-81fa-2003efe55968\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr"
Oct 31 02:14:06.340298 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.340192    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6mg8\" (UniqueName: \"kubernetes.io/projected/31f58bf2-680c-4c93-81fa-2003efe55968-kube-api-access-v6mg8\") pod \"maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr\" (UID: \"31f58bf2-680c-4c93-81fa-2003efe55968\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr"
Oct 31 02:14:06.340298 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.340295    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f58bf2-680c-4c93-81fa-2003efe55968-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr\" (UID: \"31f58bf2-680c-4c93-81fa-2003efe55968\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr"
Oct 31 02:14:06.340902 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.340875    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f58bf2-680c-4c93-81fa-2003efe55968-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr\" (UID: \"31f58bf2-680c-4c93-81fa-2003efe55968\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr"
Oct 31 02:14:06.342587 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.342565    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f58bf2-680c-4c93-81fa-2003efe55968-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr\" (UID: \"31f58bf2-680c-4c93-81fa-2003efe55968\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr"
Oct 31 02:14:06.353934 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.353903    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6mg8\" (UniqueName: \"kubernetes.io/projected/31f58bf2-680c-4c93-81fa-2003efe55968-kube-api-access-v6mg8\") pod \"maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr\" (UID: \"31f58bf2-680c-4c93-81fa-2003efe55968\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr"
Oct 31 02:14:06.363723 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.363696    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr"
Oct 31 02:14:06.652420 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.652393    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr"]
Oct 31 02:14:06.653897 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:14:06.653871    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f58bf2_680c_4c93_81fa_2003efe55968.slice/crio-74178fa9968b92313aa55f0b995efd14b1c9a9a437d517a781d867948a3f90ef WatchSource:0}: Error finding container 74178fa9968b92313aa55f0b995efd14b1c9a9a437d517a781d867948a3f90ef: Status 404 returned error can't find the container with id 74178fa9968b92313aa55f0b995efd14b1c9a9a437d517a781d867948a3f90ef
Oct 31 02:14:06.968091 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.968050    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p" podUID="7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:14:06.968091 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:06.968088    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw" podUID="35f0ddb1-3758-4a14-a2e2-2c7a928b3564" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:14:07.418158 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:07.418058    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr" event={"ID":"31f58bf2-680c-4c93-81fa-2003efe55968","Type":"ContainerStarted","Data":"f3935e5137629e4a56db7bc9e61421670dda42660ea24b865d559753ede7d8ee"}
Oct 31 02:14:07.418616 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:07.418155    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr" event={"ID":"31f58bf2-680c-4c93-81fa-2003efe55968","Type":"ContainerStarted","Data":"74178fa9968b92313aa55f0b995efd14b1c9a9a437d517a781d867948a3f90ef"}
Oct 31 02:14:07.439652 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:07.439597    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr" podStartSLOduration=1.439581201 podStartE2EDuration="1.439581201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:14:07.437882604 +0000 UTC m=+3203.441596895" watchObservedRunningTime="2026-10-31 02:14:07.439581201 +0000 UTC m=+3203.443295492"
Oct 31 02:14:08.430751 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:08.430713    5650 generic.go:356] "Generic (PLEG): container finished" podID="31f58bf2-680c-4c93-81fa-2003efe55968" containerID="f3935e5137629e4a56db7bc9e61421670dda42660ea24b865d559753ede7d8ee" exitCode=0
Oct 31 02:14:08.431253 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:08.430884    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr" event={"ID":"31f58bf2-680c-4c93-81fa-2003efe55968","Type":"ContainerDied","Data":"f3935e5137629e4a56db7bc9e61421670dda42660ea24b865d559753ede7d8ee"}
Oct 31 02:14:09.622040 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:09.621975    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr"
Oct 31 02:14:09.724497 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:09.724461    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31f58bf2-680c-4c93-81fa-2003efe55968-kube-api-access-v6mg8\" (UniqueName: \"kubernetes.io/projected/31f58bf2-680c-4c93-81fa-2003efe55968-kube-api-access-v6mg8\") pod \"31f58bf2-680c-4c93-81fa-2003efe55968\" (UID: \"31f58bf2-680c-4c93-81fa-2003efe55968\") "
Oct 31 02:14:09.724738 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:09.724639    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31f58bf2-680c-4c93-81fa-2003efe55968-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f58bf2-680c-4c93-81fa-2003efe55968-maas-api-openshift-ca\") pod \"31f58bf2-680c-4c93-81fa-2003efe55968\" (UID: \"31f58bf2-680c-4c93-81fa-2003efe55968\") "
Oct 31 02:14:09.724788 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:09.724739    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31f58bf2-680c-4c93-81fa-2003efe55968-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f58bf2-680c-4c93-81fa-2003efe55968-maas-api-xks-ca\") pod \"31f58bf2-680c-4c93-81fa-2003efe55968\" (UID: \"31f58bf2-680c-4c93-81fa-2003efe55968\") "
Oct 31 02:14:09.725172 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:09.725140    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f58bf2-680c-4c93-81fa-2003efe55968-maas-api-openshift-ca" pod "31f58bf2-680c-4c93-81fa-2003efe55968" (UID: "31f58bf2-680c-4c93-81fa-2003efe55968"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:14:09.725608 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:09.725582    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f58bf2-680c-4c93-81fa-2003efe55968-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:09.726978 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:09.726951    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31f58bf2-680c-4c93-81fa-2003efe55968-maas-api-xks-ca" pod "31f58bf2-680c-4c93-81fa-2003efe55968" (UID: "31f58bf2-680c-4c93-81fa-2003efe55968"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:14:09.727312 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:09.727294    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f58bf2-680c-4c93-81fa-2003efe55968-kube-api-access-v6mg8" pod "31f58bf2-680c-4c93-81fa-2003efe55968" (UID: "31f58bf2-680c-4c93-81fa-2003efe55968"). InnerVolumeSpecName "kube-api-access-v6mg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:09.827318 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:09.827274    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6mg8\" (UniqueName: \"kubernetes.io/projected/31f58bf2-680c-4c93-81fa-2003efe55968-kube-api-access-v6mg8\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:09.827318 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:09.827310    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f58bf2-680c-4c93-81fa-2003efe55968-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:10.493488 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:10.492088    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr" event={"ID":"31f58bf2-680c-4c93-81fa-2003efe55968","Type":"ContainerDied","Data":"74178fa9968b92313aa55f0b995efd14b1c9a9a437d517a781d867948a3f90ef"}
Oct 31 02:14:10.493488 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:10.492153    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74178fa9968b92313aa55f0b995efd14b1c9a9a437d517a781d867948a3f90ef"
Oct 31 02:14:10.493488 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:10.492226    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr"
Oct 31 02:14:14.843356 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:14.843225    5650 generic.go:356] "Generic (PLEG): container finished" podID="7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e" containerID="b07ae836aa64d66b7686b5255f24807f93e13ddb1686c70b9f39d6e698af38ad" exitCode=0
Oct 31 02:14:14.843909 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:14.843519    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p" event={"ID":"7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e","Type":"ContainerDied","Data":"b07ae836aa64d66b7686b5255f24807f93e13ddb1686c70b9f39d6e698af38ad"}
Oct 31 02:14:15.101497 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.101425    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p"
Oct 31 02:14:15.174780 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.174742    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw"
Oct 31 02:14:15.259404 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.259365    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35f0ddb1-3758-4a14-a2e2-2c7a928b3564-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35f0ddb1-3758-4a14-a2e2-2c7a928b3564-plugins-config-volume\") pod \"35f0ddb1-3758-4a14-a2e2-2c7a928b3564\" (UID: \"35f0ddb1-3758-4a14-a2e2-2c7a928b3564\") "
Oct 31 02:14:15.259661 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.259483    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e-plugins-config-volume\") pod \"7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e\" (UID: \"7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e\") "
Oct 31 02:14:15.259661 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.259517    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f0ddb1-3758-4a14-a2e2-2c7a928b3564-kube-api-access-6pkl4\" (UniqueName: \"kubernetes.io/projected/35f0ddb1-3758-4a14-a2e2-2c7a928b3564-kube-api-access-6pkl4\") pod \"35f0ddb1-3758-4a14-a2e2-2c7a928b3564\" (UID: \"35f0ddb1-3758-4a14-a2e2-2c7a928b3564\") "
Oct 31 02:14:15.259661 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.259581    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e-kube-api-access-xcfcz\" (UniqueName: \"kubernetes.io/projected/7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e-kube-api-access-xcfcz\") pod \"7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e\" (UID: \"7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e\") "
Oct 31 02:14:15.259827 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.259810    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35f0ddb1-3758-4a14-a2e2-2c7a928b3564-plugins-config-volume" pod "35f0ddb1-3758-4a14-a2e2-2c7a928b3564" (UID: "35f0ddb1-3758-4a14-a2e2-2c7a928b3564"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:14:15.259944 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.259916    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e-plugins-config-volume" pod "7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e" (UID: "7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:14:15.260758 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.260695    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35f0ddb1-3758-4a14-a2e2-2c7a928b3564-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:15.260922 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.260767    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:15.262133 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.262111    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e-kube-api-access-xcfcz" pod "7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e" (UID: "7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e"). InnerVolumeSpecName "kube-api-access-xcfcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:15.262457 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.262429    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f0ddb1-3758-4a14-a2e2-2c7a928b3564-kube-api-access-6pkl4" pod "35f0ddb1-3758-4a14-a2e2-2c7a928b3564" (UID: "35f0ddb1-3758-4a14-a2e2-2c7a928b3564"). InnerVolumeSpecName "kube-api-access-6pkl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:15.362962 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.362867    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pkl4\" (UniqueName: \"kubernetes.io/projected/35f0ddb1-3758-4a14-a2e2-2c7a928b3564-kube-api-access-6pkl4\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:15.362962 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.362908    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcfcz\" (UniqueName: \"kubernetes.io/projected/7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e-kube-api-access-xcfcz\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:15.853268 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.853222    5650 generic.go:356] "Generic (PLEG): container finished" podID="35f0ddb1-3758-4a14-a2e2-2c7a928b3564" containerID="97d4a78fcd99709176951137a6a5e07e325bafa066bb7663b8021cd346be6e69" exitCode=0
Oct 31 02:14:15.853268 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.853270    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw"
Oct 31 02:14:15.853876 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.853286    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw" event={"ID":"35f0ddb1-3758-4a14-a2e2-2c7a928b3564","Type":"ContainerDied","Data":"97d4a78fcd99709176951137a6a5e07e325bafa066bb7663b8021cd346be6e69"}
Oct 31 02:14:15.853876 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.853343    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw" event={"ID":"35f0ddb1-3758-4a14-a2e2-2c7a928b3564","Type":"ContainerDied","Data":"b6cd9586cc5f6a2d4b3cbd14af686d3597b9bc7e3df712dcdf85a137dc3f47c7"}
Oct 31 02:14:15.853876 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.853367    5650 scope.go:122] "RemoveContainer" containerID="97d4a78fcd99709176951137a6a5e07e325bafa066bb7663b8021cd346be6e69"
Oct 31 02:14:15.855364 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.855343    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p" event={"ID":"7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e","Type":"ContainerDied","Data":"5d215e0797b6ea88bb95117911e3ed0cdeb597518a80d533b8db5293445831d7"}
Oct 31 02:14:15.855485 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.855364    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p"
Oct 31 02:14:15.867234 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.867211    5650 scope.go:122] "RemoveContainer" containerID="97d4a78fcd99709176951137a6a5e07e325bafa066bb7663b8021cd346be6e69"
Oct 31 02:14:15.867578 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:14:15.867559    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97d4a78fcd99709176951137a6a5e07e325bafa066bb7663b8021cd346be6e69\": container with ID starting with 97d4a78fcd99709176951137a6a5e07e325bafa066bb7663b8021cd346be6e69 not found: ID does not exist" containerID="97d4a78fcd99709176951137a6a5e07e325bafa066bb7663b8021cd346be6e69"
Oct 31 02:14:15.867631 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.867588    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97d4a78fcd99709176951137a6a5e07e325bafa066bb7663b8021cd346be6e69"} err="failed to get container status \"97d4a78fcd99709176951137a6a5e07e325bafa066bb7663b8021cd346be6e69\": rpc error: code = NotFound desc = could not find container \"97d4a78fcd99709176951137a6a5e07e325bafa066bb7663b8021cd346be6e69\": container with ID starting with 97d4a78fcd99709176951137a6a5e07e325bafa066bb7663b8021cd346be6e69 not found: ID does not exist"
Oct 31 02:14:15.867631 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.867604    5650 scope.go:122] "RemoveContainer" containerID="b07ae836aa64d66b7686b5255f24807f93e13ddb1686c70b9f39d6e698af38ad"
Oct 31 02:14:15.929742 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.929689    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw"]
Oct 31 02:14:15.935803 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.935768    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-be6935-b7b86bd59-9sdpw"]
Oct 31 02:14:15.945880 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.945847    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p"]
Oct 31 02:14:15.950429 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:15.950400    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-be6935-7cdc4bc59-wtf9p"]
Oct 31 02:14:16.121646 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:16.121567    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35f0ddb1-3758-4a14-a2e2-2c7a928b3564" path="/var/lib/kubelet/pods/35f0ddb1-3758-4a14-a2e2-2c7a928b3564/volumes"
Oct 31 02:14:16.122035 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:16.122021    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e" path="/var/lib/kubelet/pods/7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e/volumes"
Oct 31 02:14:16.179828 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:16.179707    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb"]
Oct 31 02:14:16.186355 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:16.186317    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv"]
Oct 31 02:14:21.182802 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:21.182748    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb" podUID="652c2106-a19b-4498-9ed2-c3775bcca730" containerName="payload-pre-processing" containerID="cri-o://001ad07148689e3e3c9d2e2eb4070ae14fe958561126a3a68cff692cf57775ec" gracePeriod=25
Oct 31 02:14:21.187055 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:21.187016    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv" podUID="5d87a817-5974-457b-b870-45ce974b5148" containerName="payload-processing" containerID="cri-o://950f0614ddc7369253b40b7313eaa18a36741a47ed6708b34501089589a97f20" gracePeriod=25
Oct 31 02:14:28.564470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:28.563535    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"]
Oct 31 02:14:28.564470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:28.563813    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr" podUID="94727d94-112b-437f-84f5-bffde7e46284" containerName="istio-proxy" containerID="cri-o://43d5f1809c05ccb9a4e29c496e5947caabc61b78643e9ff8b82a085a631411ab" gracePeriod=30
Oct 31 02:14:28.742472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:28.741533    5650 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:14:28.753823 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:28.753786    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"]
Oct 31 02:14:28.759498 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:28.759412    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d" podUID="d98cd19e-f8d9-4e07-b319-be194ea8f935" containerName="istio-proxy" containerID="cri-o://402cb2a9e2e4de397a8dbbffffe55ca32740b06e8db69fa0998e1072a6bd0c3e" gracePeriod=30
Oct 31 02:14:28.786569 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:28.786535    5650 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:14:28.786857 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:28.786829    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d" podUID="d98cd19e-f8d9-4e07-b319-be194ea8f935" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:14:28.796502 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:14:28.796462    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:14:28.824527 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:28.824125    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb" podUID="652c2106-a19b-4498-9ed2-c3775bcca730" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:14:29.860526 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:29.860130    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv" podUID="5d87a817-5974-457b-b870-45ce974b5148" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:14:32.126851 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:32.126816    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr"]
Oct 31 02:14:32.126851 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:32.126856    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ac7e0d-n6vjr"]
Oct 31 02:14:34.094827 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.094798    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 02:14:34.098663 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.098641    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 02:14:34.119845 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.119816    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31f58bf2-680c-4c93-81fa-2003efe55968" path="/var/lib/kubelet/pods/31f58bf2-680c-4c93-81fa-2003efe55968/volumes"
Oct 31 02:14:34.178084 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.178047    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-workload-certs\") pod \"d98cd19e-f8d9-4e07-b319-be194ea8f935\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") "
Oct 31 02:14:34.178455 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.178409    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-workload-certs" pod "d98cd19e-f8d9-4e07-b319-be194ea8f935" (UID: "d98cd19e-f8d9-4e07-b319-be194ea8f935"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:34.179111 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.178980    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.267929 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.267890    5650 generic.go:356] "Generic (PLEG): container finished" podID="d98cd19e-f8d9-4e07-b319-be194ea8f935" containerID="402cb2a9e2e4de397a8dbbffffe55ca32740b06e8db69fa0998e1072a6bd0c3e" exitCode=0
Oct 31 02:14:34.268183 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.267955    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"
Oct 31 02:14:34.268183 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.268013    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d" event={"ID":"d98cd19e-f8d9-4e07-b319-be194ea8f935","Type":"ContainerDied","Data":"402cb2a9e2e4de397a8dbbffffe55ca32740b06e8db69fa0998e1072a6bd0c3e"}
Oct 31 02:14:34.268183 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.268063    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d" event={"ID":"d98cd19e-f8d9-4e07-b319-be194ea8f935","Type":"ContainerDied","Data":"1625fcf923c143a7269bfa8863463cb94fd6abdef87bc607bc950a42ddb2159a"}
Oct 31 02:14:34.268183 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.268091    5650 scope.go:122] "RemoveContainer" containerID="402cb2a9e2e4de397a8dbbffffe55ca32740b06e8db69fa0998e1072a6bd0c3e"
Oct 31 02:14:34.270653 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.270619    5650 generic.go:356] "Generic (PLEG): container finished" podID="94727d94-112b-437f-84f5-bffde7e46284" containerID="43d5f1809c05ccb9a4e29c496e5947caabc61b78643e9ff8b82a085a631411ab" exitCode=0
Oct 31 02:14:34.270812 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.270691    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"
Oct 31 02:14:34.271417 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.270790    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr" event={"ID":"94727d94-112b-437f-84f5-bffde7e46284","Type":"ContainerDied","Data":"43d5f1809c05ccb9a4e29c496e5947caabc61b78643e9ff8b82a085a631411ab"}
Oct 31 02:14:34.271417 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.270873    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr" event={"ID":"94727d94-112b-437f-84f5-bffde7e46284","Type":"ContainerDied","Data":"da3f6641da6d1faccbf5f049a0a9ce378167e1a3aeb299a3c6a8c86abc632835"}
Oct 31 02:14:34.279834 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.279765    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d98cd19e-f8d9-4e07-b319-be194ea8f935-kube-api-access-788n8\" (UniqueName: \"kubernetes.io/projected/d98cd19e-f8d9-4e07-b319-be194ea8f935-kube-api-access-788n8\") pod \"d98cd19e-f8d9-4e07-b319-be194ea8f935\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") "
Oct 31 02:14:34.279834 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.279828    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94727d94-112b-437f-84f5-bffde7e46284-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94727d94-112b-437f-84f5-bffde7e46284-istio-podinfo\") pod \"94727d94-112b-437f-84f5-bffde7e46284\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") "
Oct 31 02:14:34.280062 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.279863    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-credential-socket\") pod \"94727d94-112b-437f-84f5-bffde7e46284\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") "
Oct 31 02:14:34.280062 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.279944    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-workload-socket\") pod \"d98cd19e-f8d9-4e07-b319-be194ea8f935\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") "
Oct 31 02:14:34.280175 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.280102    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d98cd19e-f8d9-4e07-b319-be194ea8f935-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98cd19e-f8d9-4e07-b319-be194ea8f935-istiod-ca-cert\") pod \"d98cd19e-f8d9-4e07-b319-be194ea8f935\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") "
Oct 31 02:14:34.280225 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.280171    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-credential-socket\") pod \"d98cd19e-f8d9-4e07-b319-be194ea8f935\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") "
Oct 31 02:14:34.280652 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.280627    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94727d94-112b-437f-84f5-bffde7e46284-kube-api-access-9jhvq\" (UniqueName: \"kubernetes.io/projected/94727d94-112b-437f-84f5-bffde7e46284-kube-api-access-9jhvq\") pod \"94727d94-112b-437f-84f5-bffde7e46284\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") "
Oct 31 02:14:34.280766 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.280693    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-podinfo\") pod \"d98cd19e-f8d9-4e07-b319-be194ea8f935\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") "
Oct 31 02:14:34.280766 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.280724    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-token\" (UniqueName: \"kubernetes.io/projected/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-token\") pod \"d98cd19e-f8d9-4e07-b319-be194ea8f935\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") "
Oct 31 02:14:34.280766 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.280762    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-envoy\") pod \"d98cd19e-f8d9-4e07-b319-be194ea8f935\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") "
Oct 31 02:14:34.280869 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.280795    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-istio-envoy\") pod \"94727d94-112b-437f-84f5-bffde7e46284\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") "
Oct 31 02:14:34.280869 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.280816    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-istio-data\") pod \"94727d94-112b-437f-84f5-bffde7e46284\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") "
Oct 31 02:14:34.280869 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.280841    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94727d94-112b-437f-84f5-bffde7e46284-istio-token\" (UniqueName: \"kubernetes.io/projected/94727d94-112b-437f-84f5-bffde7e46284-istio-token\") pod \"94727d94-112b-437f-84f5-bffde7e46284\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") "
Oct 31 02:14:34.280972 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.280898    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-workload-certs\") pod \"94727d94-112b-437f-84f5-bffde7e46284\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") "
Oct 31 02:14:34.280972 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.280931    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94727d94-112b-437f-84f5-bffde7e46284-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94727d94-112b-437f-84f5-bffde7e46284-istiod-ca-cert\") pod \"94727d94-112b-437f-84f5-bffde7e46284\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") "
Oct 31 02:14:34.281069 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.281012    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-data\") pod \"d98cd19e-f8d9-4e07-b319-be194ea8f935\" (UID: \"d98cd19e-f8d9-4e07-b319-be194ea8f935\") "
Oct 31 02:14:34.281069 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.281054    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-workload-socket\") pod \"94727d94-112b-437f-84f5-bffde7e46284\" (UID: \"94727d94-112b-437f-84f5-bffde7e46284\") "
Oct 31 02:14:34.282013 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.281989    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-workload-socket" pod "94727d94-112b-437f-84f5-bffde7e46284" (UID: "94727d94-112b-437f-84f5-bffde7e46284"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:34.284107 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.283898    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-credential-socket" pod "94727d94-112b-437f-84f5-bffde7e46284" (UID: "94727d94-112b-437f-84f5-bffde7e46284"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:34.284498 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.284290    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-workload-socket" pod "d98cd19e-f8d9-4e07-b319-be194ea8f935" (UID: "d98cd19e-f8d9-4e07-b319-be194ea8f935"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:34.284838 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.284708    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-workload-certs" pod "94727d94-112b-437f-84f5-bffde7e46284" (UID: "94727d94-112b-437f-84f5-bffde7e46284"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:34.284838 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.284739    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98cd19e-f8d9-4e07-b319-be194ea8f935-kube-api-access-788n8" pod "d98cd19e-f8d9-4e07-b319-be194ea8f935" (UID: "d98cd19e-f8d9-4e07-b319-be194ea8f935"). InnerVolumeSpecName "kube-api-access-788n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:34.284838 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.284760    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d98cd19e-f8d9-4e07-b319-be194ea8f935-istiod-ca-cert" pod "d98cd19e-f8d9-4e07-b319-be194ea8f935" (UID: "d98cd19e-f8d9-4e07-b319-be194ea8f935"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:14:34.285063 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.284861    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-istio-data" pod "94727d94-112b-437f-84f5-bffde7e46284" (UID: "94727d94-112b-437f-84f5-bffde7e46284"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:34.285345 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.285218    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-envoy" pod "d98cd19e-f8d9-4e07-b319-be194ea8f935" (UID: "d98cd19e-f8d9-4e07-b319-be194ea8f935"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:34.285345 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.285222    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94727d94-112b-437f-84f5-bffde7e46284-istiod-ca-cert" pod "94727d94-112b-437f-84f5-bffde7e46284" (UID: "94727d94-112b-437f-84f5-bffde7e46284"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:14:34.285970 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.285562    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-data" pod "d98cd19e-f8d9-4e07-b319-be194ea8f935" (UID: "d98cd19e-f8d9-4e07-b319-be194ea8f935"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:34.285970 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.285642    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-credential-socket" pod "d98cd19e-f8d9-4e07-b319-be194ea8f935" (UID: "d98cd19e-f8d9-4e07-b319-be194ea8f935"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:34.287263 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.287239    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-token" pod "d98cd19e-f8d9-4e07-b319-be194ea8f935" (UID: "d98cd19e-f8d9-4e07-b319-be194ea8f935"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:34.287783 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.287729    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94727d94-112b-437f-84f5-bffde7e46284-kube-api-access-9jhvq" pod "94727d94-112b-437f-84f5-bffde7e46284" (UID: "94727d94-112b-437f-84f5-bffde7e46284"). InnerVolumeSpecName "kube-api-access-9jhvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:34.288869 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.288843    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94727d94-112b-437f-84f5-bffde7e46284-istio-token" pod "94727d94-112b-437f-84f5-bffde7e46284" (UID: "94727d94-112b-437f-84f5-bffde7e46284"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:34.289293 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.289263    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-istio-envoy" pod "94727d94-112b-437f-84f5-bffde7e46284" (UID: "94727d94-112b-437f-84f5-bffde7e46284"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:14:34.289394 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.289367    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-podinfo" pod "d98cd19e-f8d9-4e07-b319-be194ea8f935" (UID: "d98cd19e-f8d9-4e07-b319-be194ea8f935"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:14:34.289574 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.289550    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94727d94-112b-437f-84f5-bffde7e46284-istio-podinfo" pod "94727d94-112b-437f-84f5-bffde7e46284" (UID: "94727d94-112b-437f-84f5-bffde7e46284"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:14:34.290366 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.290341    5650 scope.go:122] "RemoveContainer" containerID="402cb2a9e2e4de397a8dbbffffe55ca32740b06e8db69fa0998e1072a6bd0c3e"
Oct 31 02:14:34.290833 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:14:34.290810    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"402cb2a9e2e4de397a8dbbffffe55ca32740b06e8db69fa0998e1072a6bd0c3e\": container with ID starting with 402cb2a9e2e4de397a8dbbffffe55ca32740b06e8db69fa0998e1072a6bd0c3e not found: ID does not exist" containerID="402cb2a9e2e4de397a8dbbffffe55ca32740b06e8db69fa0998e1072a6bd0c3e"
Oct 31 02:14:34.290915 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.290842    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"402cb2a9e2e4de397a8dbbffffe55ca32740b06e8db69fa0998e1072a6bd0c3e"} err="failed to get container status \"402cb2a9e2e4de397a8dbbffffe55ca32740b06e8db69fa0998e1072a6bd0c3e\": rpc error: code = NotFound desc = could not find container \"402cb2a9e2e4de397a8dbbffffe55ca32740b06e8db69fa0998e1072a6bd0c3e\": container with ID starting with 402cb2a9e2e4de397a8dbbffffe55ca32740b06e8db69fa0998e1072a6bd0c3e not found: ID does not exist"
Oct 31 02:14:34.290915 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.290862    5650 scope.go:122] "RemoveContainer" containerID="43d5f1809c05ccb9a4e29c496e5947caabc61b78643e9ff8b82a085a631411ab"
Oct 31 02:14:34.308900 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.308875    5650 scope.go:122] "RemoveContainer" containerID="43d5f1809c05ccb9a4e29c496e5947caabc61b78643e9ff8b82a085a631411ab"
Oct 31 02:14:34.309309 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:14:34.309277    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43d5f1809c05ccb9a4e29c496e5947caabc61b78643e9ff8b82a085a631411ab\": container with ID starting with 43d5f1809c05ccb9a4e29c496e5947caabc61b78643e9ff8b82a085a631411ab not found: ID does not exist" containerID="43d5f1809c05ccb9a4e29c496e5947caabc61b78643e9ff8b82a085a631411ab"
Oct 31 02:14:34.309422 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.309320    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d5f1809c05ccb9a4e29c496e5947caabc61b78643e9ff8b82a085a631411ab"} err="failed to get container status \"43d5f1809c05ccb9a4e29c496e5947caabc61b78643e9ff8b82a085a631411ab\": rpc error: code = NotFound desc = could not find container \"43d5f1809c05ccb9a4e29c496e5947caabc61b78643e9ff8b82a085a631411ab\": container with ID starting with 43d5f1809c05ccb9a4e29c496e5947caabc61b78643e9ff8b82a085a631411ab not found: ID does not exist"
Oct 31 02:14:34.386142 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.386092    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94727d94-112b-437f-84f5-bffde7e46284-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.386142 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.386134    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.386142 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.386151    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.386582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.386168    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d98cd19e-f8d9-4e07-b319-be194ea8f935-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.386582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.386182    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.386582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.386196    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jhvq\" (UniqueName: \"kubernetes.io/projected/94727d94-112b-437f-84f5-bffde7e46284-kube-api-access-9jhvq\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.386582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.386211    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.386582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.386225    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.386582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.386244    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.386582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.386260    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.386582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.386274    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.386582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.386291    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94727d94-112b-437f-84f5-bffde7e46284-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.386582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.386307    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.386582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.386324    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94727d94-112b-437f-84f5-bffde7e46284-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.386582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.386343    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d98cd19e-f8d9-4e07-b319-be194ea8f935-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.386582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.386366    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94727d94-112b-437f-84f5-bffde7e46284-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.386582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.386386    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-788n8\" (UniqueName: \"kubernetes.io/projected/d98cd19e-f8d9-4e07-b319-be194ea8f935-kube-api-access-788n8\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:34.600173 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.600139    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"]
Oct 31 02:14:34.608012 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.607980    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-zj42d"]
Oct 31 02:14:34.629851 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.629804    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"]
Oct 31 02:14:34.633968 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:34.633935    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-glmvr"]
Oct 31 02:14:36.121230 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:36.121197    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94727d94-112b-437f-84f5-bffde7e46284" path="/var/lib/kubelet/pods/94727d94-112b-437f-84f5-bffde7e46284/volumes"
Oct 31 02:14:36.122183 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:36.122165    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d98cd19e-f8d9-4e07-b319-be194ea8f935" path="/var/lib/kubelet/pods/d98cd19e-f8d9-4e07-b319-be194ea8f935/volumes"
Oct 31 02:14:38.823593 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:38.823547    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb" podUID="652c2106-a19b-4498-9ed2-c3775bcca730" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:14:39.881466 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:39.878943    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv" podUID="5d87a817-5974-457b-b870-45ce974b5148" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:14:41.343620 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:41.343340    5650 generic.go:356] "Generic (PLEG): container finished" podID="652c2106-a19b-4498-9ed2-c3775bcca730" containerID="001ad07148689e3e3c9d2e2eb4070ae14fe958561126a3a68cff692cf57775ec" exitCode=0
Oct 31 02:14:41.343620 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:41.343406    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb" event={"ID":"652c2106-a19b-4498-9ed2-c3775bcca730","Type":"ContainerDied","Data":"001ad07148689e3e3c9d2e2eb4070ae14fe958561126a3a68cff692cf57775ec"}
Oct 31 02:14:41.350126 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:41.348401    5650 generic.go:356] "Generic (PLEG): container finished" podID="5d87a817-5974-457b-b870-45ce974b5148" containerID="950f0614ddc7369253b40b7313eaa18a36741a47ed6708b34501089589a97f20" exitCode=0
Oct 31 02:14:41.350126 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:41.348562    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv" event={"ID":"5d87a817-5974-457b-b870-45ce974b5148","Type":"ContainerDied","Data":"950f0614ddc7369253b40b7313eaa18a36741a47ed6708b34501089589a97f20"}
Oct 31 02:14:41.423064 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:41.423036    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb"
Oct 31 02:14:41.512892 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:41.512861    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv"
Oct 31 02:14:41.569494 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:41.569422    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/652c2106-a19b-4498-9ed2-c3775bcca730-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/652c2106-a19b-4498-9ed2-c3775bcca730-plugins-config-volume\") pod \"652c2106-a19b-4498-9ed2-c3775bcca730\" (UID: \"652c2106-a19b-4498-9ed2-c3775bcca730\") "
Oct 31 02:14:41.569703 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:41.569627    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/652c2106-a19b-4498-9ed2-c3775bcca730-kube-api-access-wdhz9\" (UniqueName: \"kubernetes.io/projected/652c2106-a19b-4498-9ed2-c3775bcca730-kube-api-access-wdhz9\") pod \"652c2106-a19b-4498-9ed2-c3775bcca730\" (UID: \"652c2106-a19b-4498-9ed2-c3775bcca730\") "
Oct 31 02:14:41.569945 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:41.569915    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/652c2106-a19b-4498-9ed2-c3775bcca730-plugins-config-volume" pod "652c2106-a19b-4498-9ed2-c3775bcca730" (UID: "652c2106-a19b-4498-9ed2-c3775bcca730"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:14:41.570523 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:41.570494    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/652c2106-a19b-4498-9ed2-c3775bcca730-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:41.571881 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:41.571859    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652c2106-a19b-4498-9ed2-c3775bcca730-kube-api-access-wdhz9" pod "652c2106-a19b-4498-9ed2-c3775bcca730" (UID: "652c2106-a19b-4498-9ed2-c3775bcca730"). InnerVolumeSpecName "kube-api-access-wdhz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:41.674474 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:41.671117    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d87a817-5974-457b-b870-45ce974b5148-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d87a817-5974-457b-b870-45ce974b5148-plugins-config-volume\") pod \"5d87a817-5974-457b-b870-45ce974b5148\" (UID: \"5d87a817-5974-457b-b870-45ce974b5148\") "
Oct 31 02:14:41.674474 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:41.671474    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d87a817-5974-457b-b870-45ce974b5148-kube-api-access-dxnxq\" (UniqueName: \"kubernetes.io/projected/5d87a817-5974-457b-b870-45ce974b5148-kube-api-access-dxnxq\") pod \"5d87a817-5974-457b-b870-45ce974b5148\" (UID: \"5d87a817-5974-457b-b870-45ce974b5148\") "
Oct 31 02:14:41.689591 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:41.688845    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d87a817-5974-457b-b870-45ce974b5148-plugins-config-volume" pod "5d87a817-5974-457b-b870-45ce974b5148" (UID: "5d87a817-5974-457b-b870-45ce974b5148"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:14:41.689803 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:41.689774    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d87a817-5974-457b-b870-45ce974b5148-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:41.689875 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:41.689806    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdhz9\" (UniqueName: \"kubernetes.io/projected/652c2106-a19b-4498-9ed2-c3775bcca730-kube-api-access-wdhz9\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:41.694473 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:41.694088    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d87a817-5974-457b-b870-45ce974b5148-kube-api-access-dxnxq" pod "5d87a817-5974-457b-b870-45ce974b5148" (UID: "5d87a817-5974-457b-b870-45ce974b5148"). InnerVolumeSpecName "kube-api-access-dxnxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:14:41.791141 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:41.791051    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxnxq\" (UniqueName: \"kubernetes.io/projected/5d87a817-5974-457b-b870-45ce974b5148-kube-api-access-dxnxq\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:14:42.390836 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:42.390755    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv"
Oct 31 02:14:42.390836 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:42.390769    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv" event={"ID":"5d87a817-5974-457b-b870-45ce974b5148","Type":"ContainerDied","Data":"3f3ed209b5bf4e502fc8eab5f1abb2f453f8af098af1bffb17f28f7da41b4ba2"}
Oct 31 02:14:42.391350 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:42.390842    5650 scope.go:122] "RemoveContainer" containerID="950f0614ddc7369253b40b7313eaa18a36741a47ed6708b34501089589a97f20"
Oct 31 02:14:42.392743 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:42.392713    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb" event={"ID":"652c2106-a19b-4498-9ed2-c3775bcca730","Type":"ContainerDied","Data":"dd8c7cc0b87ad58902c4b50c969ccb27ff242d134718c3f86071d13b2c61ac55"}
Oct 31 02:14:42.392887 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:42.392763    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb"
Oct 31 02:14:42.403609 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:42.403583    5650 scope.go:122] "RemoveContainer" containerID="001ad07148689e3e3c9d2e2eb4070ae14fe958561126a3a68cff692cf57775ec"
Oct 31 02:14:42.413177 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:42.413145    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb"]
Oct 31 02:14:42.418833 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:42.418808    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ac7e0d-7b9cf656bc-j7cvb"]
Oct 31 02:14:42.432409 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:42.432377    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv"]
Oct 31 02:14:42.438071 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:42.438042    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ac7e0d-97dbbb8ff-vskpv"]
Oct 31 02:14:44.122219 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:44.122189    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d87a817-5974-457b-b870-45ce974b5148" path="/var/lib/kubelet/pods/5d87a817-5974-457b-b870-45ce974b5148/volumes"
Oct 31 02:14:44.122668 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:44.122648    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="652c2106-a19b-4498-9ed2-c3775bcca730" path="/var/lib/kubelet/pods/652c2106-a19b-4498-9ed2-c3775bcca730/volumes"
Oct 31 02:14:48.327384 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:48.327349    5650 scope.go:122] "RemoveContainer" containerID="375a7127ab9ea2a9a5b09dfe0a7c72b5784c14de39e8011f36e5f94b714b9a70"
Oct 31 02:14:50.395878 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:50.395839    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2"]
Oct 31 02:14:55.396668 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:14:55.396568    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2" podUID="6f0840a2-aff1-4bb6-bba8-968c0ee0e277" containerName="payload-pre-processing" containerID="cri-o://01ac5716fee527fe7ee4d5017bf280005ff763c34f1969cbc2098feed6624dfd" gracePeriod=25
Oct 31 02:14:58.557266 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:14:58.557224    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:15:00.155241 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.155200    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7"]
Oct 31 02:15:00.156424 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.156401    5650 cpu_manager.go:414] "containerMap: removing container" podUID="d98cd19e-f8d9-4e07-b319-be194ea8f935" containerName="istio-proxy"
Oct 31 02:15:00.156491 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.156427    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98cd19e-f8d9-4e07-b319-be194ea8f935" containerName="istio-proxy"
Oct 31 02:15:00.156491 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.156462    5650 cpu_manager.go:414] "containerMap: removing container" podUID="31f58bf2-680c-4c93-81fa-2003efe55968" containerName="revoke-keys"
Oct 31 02:15:00.156491 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.156471    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31f58bf2-680c-4c93-81fa-2003efe55968" containerName="revoke-keys"
Oct 31 02:15:00.156597 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.156501    5650 cpu_manager.go:414] "containerMap: removing container" podUID="35f0ddb1-3758-4a14-a2e2-2c7a928b3564" containerName="payload-pre-processing"
Oct 31 02:15:00.156597 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.156510    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35f0ddb1-3758-4a14-a2e2-2c7a928b3564" containerName="payload-pre-processing"
Oct 31 02:15:00.156597 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.156526    5650 cpu_manager.go:414] "containerMap: removing container" podUID="7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e" containerName="payload-processing"
Oct 31 02:15:00.156597 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.156534    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e" containerName="payload-processing"
Oct 31 02:15:00.156597 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.156551    5650 cpu_manager.go:414] "containerMap: removing container" podUID="652c2106-a19b-4498-9ed2-c3775bcca730" containerName="payload-pre-processing"
Oct 31 02:15:00.156597 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.156559    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="652c2106-a19b-4498-9ed2-c3775bcca730" containerName="payload-pre-processing"
Oct 31 02:15:00.156597 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.156576    5650 cpu_manager.go:414] "containerMap: removing container" podUID="5d87a817-5974-457b-b870-45ce974b5148" containerName="payload-processing"
Oct 31 02:15:00.156597 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.156584    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d87a817-5974-457b-b870-45ce974b5148" containerName="payload-processing"
Oct 31 02:15:00.156851 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.156624    5650 cpu_manager.go:414] "containerMap: removing container" podUID="94727d94-112b-437f-84f5-bffde7e46284" containerName="istio-proxy"
Oct 31 02:15:00.156851 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.156632    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94727d94-112b-437f-84f5-bffde7e46284" containerName="istio-proxy"
Oct 31 02:15:00.156851 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.156810    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98cd19e-f8d9-4e07-b319-be194ea8f935" containerName="istio-proxy"
Oct 31 02:15:00.156851 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.156822    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="31f58bf2-680c-4c93-81fa-2003efe55968" containerName="revoke-keys"
Oct 31 02:15:00.156851 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.156833    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="35f0ddb1-3758-4a14-a2e2-2c7a928b3564" containerName="payload-pre-processing"
Oct 31 02:15:00.156851 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.156845    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e45eafd-ca74-4a6e-b2c7-e3fd8cd15d5e" containerName="payload-processing"
Oct 31 02:15:00.157034 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.156856    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="652c2106-a19b-4498-9ed2-c3775bcca730" containerName="payload-pre-processing"
Oct 31 02:15:00.157034 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.156868    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d87a817-5974-457b-b870-45ce974b5148" containerName="payload-processing"
Oct 31 02:15:00.157034 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.156884    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="94727d94-112b-437f-84f5-bffde7e46284" containerName="istio-proxy"
Oct 31 02:15:00.157034 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.156894    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d39c057-51e1-49f1-85de-facd9e8f15fc" containerName="istio-proxy"
Oct 31 02:15:00.163694 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.163664    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84"]
Oct 31 02:15:00.165253 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.165227    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7"
Oct 31 02:15:00.171755 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.171732    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84"
Oct 31 02:15:00.174237 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.174204    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7"]
Oct 31 02:15:00.179245 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.179220    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:15:00.181155 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.181134    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84"]
Oct 31 02:15:00.186980 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.186944    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7"]
Oct 31 02:15:00.208530 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.208501    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj"]
Oct 31 02:15:00.208888 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.208652    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7"
Oct 31 02:15:00.226628 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.216270    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7"]
Oct 31 02:15:00.226628 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.216364    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj"]
Oct 31 02:15:00.226628 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.216425    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql"]
Oct 31 02:15:00.226628 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.218717    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj"
Oct 31 02:15:00.253400 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.239866    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql"]
Oct 31 02:15:00.253400 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.239983    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql"
Oct 31 02:15:00.253400 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.249795    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw"]
Oct 31 02:15:00.257639 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.257613    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2"]
Oct 31 02:15:00.267620 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.257735    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw"
Oct 31 02:15:00.280175 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.280147    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2"
Oct 31 02:15:00.280501 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.280484    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw"]
Oct 31 02:15:00.280658 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.280646    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2"]
Oct 31 02:15:00.292567 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.291962    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs"]
Oct 31 02:15:00.298671 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.298539    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs"
Oct 31 02:15:00.302916 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.302652    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx"]
Oct 31 02:15:00.306000 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.304714    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2" podUID="6f0840a2-aff1-4bb6-bba8-968c0ee0e277" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:15:00.347482 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.347434    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx"
Oct 31 02:15:00.352683 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.352638    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/495060eb-5f49-4f96-bfe7-b89ed0c8b55e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7\" (UID: \"495060eb-5f49-4f96-bfe7-b89ed0c8b55e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7"
Oct 31 02:15:00.352849 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.352690    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sctpt\" (UniqueName: \"kubernetes.io/projected/9c3ce5c1-cd13-429e-9734-916e37edfb8a-kube-api-access-sctpt\") pod \"maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7\" (UID: \"9c3ce5c1-cd13-429e-9734-916e37edfb8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7"
Oct 31 02:15:00.352849 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.352742    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wb44\" (UniqueName: \"kubernetes.io/projected/495060eb-5f49-4f96-bfe7-b89ed0c8b55e-kube-api-access-5wb44\") pod \"maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7\" (UID: \"495060eb-5f49-4f96-bfe7-b89ed0c8b55e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7"
Oct 31 02:15:00.352849 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.352778    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c3ce5c1-cd13-429e-9734-916e37edfb8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7\" (UID: \"9c3ce5c1-cd13-429e-9734-916e37edfb8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7"
Oct 31 02:15:00.353031 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.352850    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6566d82-f8bc-44f8-b4ff-21edc8e67810-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84\" (UID: \"f6566d82-f8bc-44f8-b4ff-21edc8e67810\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84"
Oct 31 02:15:00.353031 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.352980    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgs9k\" (UniqueName: \"kubernetes.io/projected/87ac2c5d-5b80-44ce-904b-713134c4b4f6-kube-api-access-hgs9k\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql\" (UID: \"87ac2c5d-5b80-44ce-904b-713134c4b4f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql"
Oct 31 02:15:00.353156 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.353071    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xw6t\" (UniqueName: \"kubernetes.io/projected/ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf-kube-api-access-9xw6t\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj\" (UID: \"ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj"
Oct 31 02:15:00.353156 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.353123    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87ac2c5d-5b80-44ce-904b-713134c4b4f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql\" (UID: \"87ac2c5d-5b80-44ce-904b-713134c4b4f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql"
Oct 31 02:15:00.353268 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.353182    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj\" (UID: \"ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj"
Oct 31 02:15:00.353268 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.353233    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq2dc\" (UniqueName: \"kubernetes.io/projected/f6566d82-f8bc-44f8-b4ff-21edc8e67810-kube-api-access-xq2dc\") pod \"maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84\" (UID: \"f6566d82-f8bc-44f8-b4ff-21edc8e67810\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84"
Oct 31 02:15:00.383409 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.383351    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs"]
Oct 31 02:15:00.485470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.477643    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8cd351-2df1-4f99-a782-e63b9aab297f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx\" (UID: \"0c8cd351-2df1-4f99-a782-e63b9aab297f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx"
Oct 31 02:15:00.485470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.477716    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq2dc\" (UniqueName: \"kubernetes.io/projected/f6566d82-f8bc-44f8-b4ff-21edc8e67810-kube-api-access-xq2dc\") pod \"maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84\" (UID: \"f6566d82-f8bc-44f8-b4ff-21edc8e67810\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84"
Oct 31 02:15:00.485470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.477829    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz4jm\" (UniqueName: \"kubernetes.io/projected/0c8cd351-2df1-4f99-a782-e63b9aab297f-kube-api-access-rz4jm\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx\" (UID: \"0c8cd351-2df1-4f99-a782-e63b9aab297f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx"
Oct 31 02:15:00.524470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.519923    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/495060eb-5f49-4f96-bfe7-b89ed0c8b55e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7\" (UID: \"495060eb-5f49-4f96-bfe7-b89ed0c8b55e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7"
Oct 31 02:15:00.524470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.519985    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sctpt\" (UniqueName: \"kubernetes.io/projected/9c3ce5c1-cd13-429e-9734-916e37edfb8a-kube-api-access-sctpt\") pod \"maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7\" (UID: \"9c3ce5c1-cd13-429e-9734-916e37edfb8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7"
Oct 31 02:15:00.524470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.520035    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfceee3a-9e84-4888-a7ff-bb5fe5b632e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs\" (UID: \"dfceee3a-9e84-4888-a7ff-bb5fe5b632e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs"
Oct 31 02:15:00.524470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.520083    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wb44\" (UniqueName: \"kubernetes.io/projected/495060eb-5f49-4f96-bfe7-b89ed0c8b55e-kube-api-access-5wb44\") pod \"maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7\" (UID: \"495060eb-5f49-4f96-bfe7-b89ed0c8b55e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7"
Oct 31 02:15:00.524470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.520118    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c3ce5c1-cd13-429e-9734-916e37edfb8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7\" (UID: \"9c3ce5c1-cd13-429e-9734-916e37edfb8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7"
Oct 31 02:15:00.524470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.520262    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6566d82-f8bc-44f8-b4ff-21edc8e67810-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84\" (UID: \"f6566d82-f8bc-44f8-b4ff-21edc8e67810\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84"
Oct 31 02:15:00.524470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.520339    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0ae7217-5d7c-4dd3-b830-7e669721ec67-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890215-sn5w2\" (UID: \"e0ae7217-5d7c-4dd3-b830-7e669721ec67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2"
Oct 31 02:15:00.524470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.520383    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw\" (UID: \"6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw"
Oct 31 02:15:00.524470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.520427    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc9hv\" (UniqueName: \"kubernetes.io/projected/6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d-kube-api-access-vc9hv\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw\" (UID: \"6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw"
Oct 31 02:15:00.524470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.520530    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8q5d\" (UniqueName: \"kubernetes.io/projected/e0ae7217-5d7c-4dd3-b830-7e669721ec67-kube-api-access-c8q5d\") pod \"maas-api-key-cleanup-29890215-sn5w2\" (UID: \"e0ae7217-5d7c-4dd3-b830-7e669721ec67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2"
Oct 31 02:15:00.524470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.520566    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgs9k\" (UniqueName: \"kubernetes.io/projected/87ac2c5d-5b80-44ce-904b-713134c4b4f6-kube-api-access-hgs9k\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql\" (UID: \"87ac2c5d-5b80-44ce-904b-713134c4b4f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql"
Oct 31 02:15:00.524470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.520611    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgfq9\" (UniqueName: \"kubernetes.io/projected/dfceee3a-9e84-4888-a7ff-bb5fe5b632e5-kube-api-access-sgfq9\") pod \"maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs\" (UID: \"dfceee3a-9e84-4888-a7ff-bb5fe5b632e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs"
Oct 31 02:15:00.524470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.520709    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xw6t\" (UniqueName: \"kubernetes.io/projected/ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf-kube-api-access-9xw6t\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj\" (UID: \"ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj"
Oct 31 02:15:00.524470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.520758    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87ac2c5d-5b80-44ce-904b-713134c4b4f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql\" (UID: \"87ac2c5d-5b80-44ce-904b-713134c4b4f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql"
Oct 31 02:15:00.524470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.520822    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj\" (UID: \"ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj"
Oct 31 02:15:00.524470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.521879    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj\" (UID: \"ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj"
Oct 31 02:15:00.524470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.523040    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/495060eb-5f49-4f96-bfe7-b89ed0c8b55e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7\" (UID: \"495060eb-5f49-4f96-bfe7-b89ed0c8b55e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7"
Oct 31 02:15:00.579539 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.566494    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6566d82-f8bc-44f8-b4ff-21edc8e67810-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84\" (UID: \"f6566d82-f8bc-44f8-b4ff-21edc8e67810\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84"
Oct 31 02:15:00.597999 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.573247    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87ac2c5d-5b80-44ce-904b-713134c4b4f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql\" (UID: \"87ac2c5d-5b80-44ce-904b-713134c4b4f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql"
Oct 31 02:15:00.634850 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.634812    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0ae7217-5d7c-4dd3-b830-7e669721ec67-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890215-sn5w2\" (UID: \"e0ae7217-5d7c-4dd3-b830-7e669721ec67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2"
Oct 31 02:15:00.635029 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.634885    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw\" (UID: \"6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw"
Oct 31 02:15:00.635029 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.634937    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc9hv\" (UniqueName: \"kubernetes.io/projected/6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d-kube-api-access-vc9hv\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw\" (UID: \"6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw"
Oct 31 02:15:00.635029 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.635011    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8q5d\" (UniqueName: \"kubernetes.io/projected/e0ae7217-5d7c-4dd3-b830-7e669721ec67-kube-api-access-c8q5d\") pod \"maas-api-key-cleanup-29890215-sn5w2\" (UID: \"e0ae7217-5d7c-4dd3-b830-7e669721ec67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2"
Oct 31 02:15:00.635241 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.635058    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgfq9\" (UniqueName: \"kubernetes.io/projected/dfceee3a-9e84-4888-a7ff-bb5fe5b632e5-kube-api-access-sgfq9\") pod \"maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs\" (UID: \"dfceee3a-9e84-4888-a7ff-bb5fe5b632e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs"
Oct 31 02:15:00.635241 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.635189    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8cd351-2df1-4f99-a782-e63b9aab297f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx\" (UID: \"0c8cd351-2df1-4f99-a782-e63b9aab297f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx"
Oct 31 02:15:00.635402 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.635300    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz4jm\" (UniqueName: \"kubernetes.io/projected/0c8cd351-2df1-4f99-a782-e63b9aab297f-kube-api-access-rz4jm\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx\" (UID: \"0c8cd351-2df1-4f99-a782-e63b9aab297f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx"
Oct 31 02:15:00.637305 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.635487    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfceee3a-9e84-4888-a7ff-bb5fe5b632e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs\" (UID: \"dfceee3a-9e84-4888-a7ff-bb5fe5b632e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs"
Oct 31 02:15:00.637305 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.636848    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw\" (UID: \"6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw"
Oct 31 02:15:00.637305 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.636946    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfceee3a-9e84-4888-a7ff-bb5fe5b632e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs\" (UID: \"dfceee3a-9e84-4888-a7ff-bb5fe5b632e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs"
Oct 31 02:15:00.638962 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.638938    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0ae7217-5d7c-4dd3-b830-7e669721ec67-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890215-sn5w2\" (UID: \"e0ae7217-5d7c-4dd3-b830-7e669721ec67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2"
Oct 31 02:15:00.640764 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.640688    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8cd351-2df1-4f99-a782-e63b9aab297f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx\" (UID: \"0c8cd351-2df1-4f99-a782-e63b9aab297f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx"
Oct 31 02:15:00.641961 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.641898    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c3ce5c1-cd13-429e-9734-916e37edfb8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7\" (UID: \"9c3ce5c1-cd13-429e-9734-916e37edfb8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7"
Oct 31 02:15:00.646974 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.646931    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wb44\" (UniqueName: \"kubernetes.io/projected/495060eb-5f49-4f96-bfe7-b89ed0c8b55e-kube-api-access-5wb44\") pod \"maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7\" (UID: \"495060eb-5f49-4f96-bfe7-b89ed0c8b55e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7"
Oct 31 02:15:00.651551 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.651521    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgs9k\" (UniqueName: \"kubernetes.io/projected/87ac2c5d-5b80-44ce-904b-713134c4b4f6-kube-api-access-hgs9k\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql\" (UID: \"87ac2c5d-5b80-44ce-904b-713134c4b4f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql"
Oct 31 02:15:00.653937 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.653912    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx"]
Oct 31 02:15:00.655096 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.655072    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xw6t\" (UniqueName: \"kubernetes.io/projected/ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf-kube-api-access-9xw6t\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj\" (UID: \"ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj"
Oct 31 02:15:00.656346 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.656316    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz4jm\" (UniqueName: \"kubernetes.io/projected/0c8cd351-2df1-4f99-a782-e63b9aab297f-kube-api-access-rz4jm\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx\" (UID: \"0c8cd351-2df1-4f99-a782-e63b9aab297f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx"
Oct 31 02:15:00.662005 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.661982    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc9hv\" (UniqueName: \"kubernetes.io/projected/6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d-kube-api-access-vc9hv\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw\" (UID: \"6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw"
Oct 31 02:15:00.662202 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.661984    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8q5d\" (UniqueName: \"kubernetes.io/projected/e0ae7217-5d7c-4dd3-b830-7e669721ec67-kube-api-access-c8q5d\") pod \"maas-api-key-cleanup-29890215-sn5w2\" (UID: \"e0ae7217-5d7c-4dd3-b830-7e669721ec67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2"
Oct 31 02:15:00.666550 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.666517    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgfq9\" (UniqueName: \"kubernetes.io/projected/dfceee3a-9e84-4888-a7ff-bb5fe5b632e5-kube-api-access-sgfq9\") pod \"maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs\" (UID: \"dfceee3a-9e84-4888-a7ff-bb5fe5b632e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs"
Oct 31 02:15:00.668440 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.668410    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq2dc\" (UniqueName: \"kubernetes.io/projected/f6566d82-f8bc-44f8-b4ff-21edc8e67810-kube-api-access-xq2dc\") pod \"maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84\" (UID: \"f6566d82-f8bc-44f8-b4ff-21edc8e67810\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84"
Oct 31 02:15:00.673475 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.670234    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sctpt\" (UniqueName: \"kubernetes.io/projected/9c3ce5c1-cd13-429e-9734-916e37edfb8a-kube-api-access-sctpt\") pod \"maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7\" (UID: \"9c3ce5c1-cd13-429e-9734-916e37edfb8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7"
Oct 31 02:15:00.676961 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.676881    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"]
Oct 31 02:15:00.677149 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.677122    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr" podUID="e0fa4248-e818-450c-bf1b-74f859787788" containerName="istio-proxy" containerID="cri-o://551e174065028f94449a6fa7da1b56f484f5e2157051a24e6a28b8e075e25d3b" gracePeriod=30
Oct 31 02:15:00.808255 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.808149    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7"
Oct 31 02:15:00.829002 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.828972    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84"
Oct 31 02:15:00.859371 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.859331    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7"
Oct 31 02:15:00.884836 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.884799    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj"
Oct 31 02:15:00.896967 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.896747    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql"
Oct 31 02:15:00.918821 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.918774    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx"
Oct 31 02:15:00.927698 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.927498    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw"
Oct 31 02:15:00.957090 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.956753    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2"
Oct 31 02:15:00.960758 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:00.960728    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs"
Oct 31 02:15:01.115102 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:15:01.115061    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c3ce5c1_cd13_429e_9734_916e37edfb8a.slice/crio-61277df181baba9c7f6b03ecaabf76e4d1b1cac1fb22d2a384b3f75d62da59d2 WatchSource:0}: Error finding container 61277df181baba9c7f6b03ecaabf76e4d1b1cac1fb22d2a384b3f75d62da59d2: Status 404 returned error can't find the container with id 61277df181baba9c7f6b03ecaabf76e4d1b1cac1fb22d2a384b3f75d62da59d2
Oct 31 02:15:01.115231 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:01.115139    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7"]
Oct 31 02:15:01.347279 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:15:01.347245    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6566d82_f8bc_44f8_b4ff_21edc8e67810.slice/crio-aaffb4716719b62f0c86f6d8045f8ccd88cb0d23361647383ee8001f525984a8 WatchSource:0}: Error finding container aaffb4716719b62f0c86f6d8045f8ccd88cb0d23361647383ee8001f525984a8: Status 404 returned error can't find the container with id aaffb4716719b62f0c86f6d8045f8ccd88cb0d23361647383ee8001f525984a8
Oct 31 02:15:01.347701 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:01.347469    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84"]
Oct 31 02:15:01.365792 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:01.365761    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs"]
Oct 31 02:15:01.371559 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:15:01.371221    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfceee3a_9e84_4888_a7ff_bb5fe5b632e5.slice/crio-30cb32fddb7409b44bbae788d613d9a45b0444e79be33b904df14f6f298013a5 WatchSource:0}: Error finding container 30cb32fddb7409b44bbae788d613d9a45b0444e79be33b904df14f6f298013a5: Status 404 returned error can't find the container with id 30cb32fddb7409b44bbae788d613d9a45b0444e79be33b904df14f6f298013a5
Oct 31 02:15:01.409345 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:15:01.409314    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod495060eb_5f49_4f96_bfe7_b89ed0c8b55e.slice/crio-15e0b990b48203a62bd5b9e94cb2296d1011fabb3790edf14370bf660c89b150 WatchSource:0}: Error finding container 15e0b990b48203a62bd5b9e94cb2296d1011fabb3790edf14370bf660c89b150: Status 404 returned error can't find the container with id 15e0b990b48203a62bd5b9e94cb2296d1011fabb3790edf14370bf660c89b150
Oct 31 02:15:01.409598 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:01.409574    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7"]
Oct 31 02:15:01.452183 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:01.452154    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql"]
Oct 31 02:15:01.452657 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:15:01.452629    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ac2c5d_5b80_44ce_904b_713134c4b4f6.slice/crio-68c8cab0b14f9bf251ea18023e536588b129d86557fcc6e4f9d10d31115f96da WatchSource:0}: Error finding container 68c8cab0b14f9bf251ea18023e536588b129d86557fcc6e4f9d10d31115f96da: Status 404 returned error can't find the container with id 68c8cab0b14f9bf251ea18023e536588b129d86557fcc6e4f9d10d31115f96da
Oct 31 02:15:01.470115 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:15:01.470078    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7b3b4d_d73c_4eb5_8590_47e7c9b71b9d.slice/crio-adaad9c8a50572651061a7cbf5d2f7523709a3a9222a1417c1cb28d9da2b72a6 WatchSource:0}: Error finding container adaad9c8a50572651061a7cbf5d2f7523709a3a9222a1417c1cb28d9da2b72a6: Status 404 returned error can't find the container with id adaad9c8a50572651061a7cbf5d2f7523709a3a9222a1417c1cb28d9da2b72a6
Oct 31 02:15:01.472083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:01.472054    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw"]
Oct 31 02:15:01.485180 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:01.485148    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj"]
Oct 31 02:15:01.496116 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:15:01.496084    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5e1b12_8ae4_4e44_a876_0e7ad00d19cf.slice/crio-114d8dfba4834c21966035728ba7ec04c8e2aa37fe143ee8466d91d910782f9d WatchSource:0}: Error finding container 114d8dfba4834c21966035728ba7ec04c8e2aa37fe143ee8466d91d910782f9d: Status 404 returned error can't find the container with id 114d8dfba4834c21966035728ba7ec04c8e2aa37fe143ee8466d91d910782f9d
Oct 31 02:15:01.512012 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:01.510993    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx"]
Oct 31 02:15:01.516721 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:01.516694    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2"]
Oct 31 02:15:01.608786 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:15:01.608709    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c8cd351_2df1_4f99_a782_e63b9aab297f.slice/crio-fda7d6a3bbe36c6dc454829e443a8c58e8e291078845f7a0dd797dcd8c835486 WatchSource:0}: Error finding container fda7d6a3bbe36c6dc454829e443a8c58e8e291078845f7a0dd797dcd8c835486: Status 404 returned error can't find the container with id fda7d6a3bbe36c6dc454829e443a8c58e8e291078845f7a0dd797dcd8c835486
Oct 31 02:15:01.612996 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:01.612968    5650 patch_prober.go:29] interesting pod/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:15:01.613101 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:01.613023    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr" podUID="e0fa4248-e818-450c-bf1b-74f859787788" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:15:01.679642 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:01.679599    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7" event={"ID":"9c3ce5c1-cd13-429e-9734-916e37edfb8a","Type":"ContainerStarted","Data":"61277df181baba9c7f6b03ecaabf76e4d1b1cac1fb22d2a384b3f75d62da59d2"}
Oct 31 02:15:01.681218 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:01.681180    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7" event={"ID":"495060eb-5f49-4f96-bfe7-b89ed0c8b55e","Type":"ContainerStarted","Data":"15e0b990b48203a62bd5b9e94cb2296d1011fabb3790edf14370bf660c89b150"}
Oct 31 02:15:01.682709 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:01.682682    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs" event={"ID":"dfceee3a-9e84-4888-a7ff-bb5fe5b632e5","Type":"ContainerStarted","Data":"30cb32fddb7409b44bbae788d613d9a45b0444e79be33b904df14f6f298013a5"}
Oct 31 02:15:01.685699 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:01.685662    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx" event={"ID":"0c8cd351-2df1-4f99-a782-e63b9aab297f","Type":"ContainerStarted","Data":"fda7d6a3bbe36c6dc454829e443a8c58e8e291078845f7a0dd797dcd8c835486"}
Oct 31 02:15:01.687597 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:01.687573    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj" event={"ID":"ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf","Type":"ContainerStarted","Data":"114d8dfba4834c21966035728ba7ec04c8e2aa37fe143ee8466d91d910782f9d"}
Oct 31 02:15:01.689535 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:01.689508    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84" event={"ID":"f6566d82-f8bc-44f8-b4ff-21edc8e67810","Type":"ContainerStarted","Data":"aaffb4716719b62f0c86f6d8045f8ccd88cb0d23361647383ee8001f525984a8"}
Oct 31 02:15:01.691171 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:01.691143    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql" event={"ID":"87ac2c5d-5b80-44ce-904b-713134c4b4f6","Type":"ContainerStarted","Data":"68c8cab0b14f9bf251ea18023e536588b129d86557fcc6e4f9d10d31115f96da"}
Oct 31 02:15:01.692727 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:01.692702    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw" event={"ID":"6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d","Type":"ContainerStarted","Data":"adaad9c8a50572651061a7cbf5d2f7523709a3a9222a1417c1cb28d9da2b72a6"}
Oct 31 02:15:02.706844 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:02.706790    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql" event={"ID":"87ac2c5d-5b80-44ce-904b-713134c4b4f6","Type":"ContainerStarted","Data":"73615ec0619cc62c7a68b0c347dc55b0361a961892031d1354cc210623d2ecbe"}
Oct 31 02:15:02.709279 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:02.709252    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw" event={"ID":"6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d","Type":"ContainerStarted","Data":"f8d81bdacf514121bf2a9727ebdcc61ec958f79720dc2da78c76eabb6879ac92"}
Oct 31 02:15:02.712183 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:02.712157    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7" event={"ID":"9c3ce5c1-cd13-429e-9734-916e37edfb8a","Type":"ContainerStarted","Data":"4749283eb928c88d4bfb36535c3457c745e17d9532521e7dc7e6d3f1dbb689a9"}
Oct 31 02:15:02.714219 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:02.714192    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7" event={"ID":"495060eb-5f49-4f96-bfe7-b89ed0c8b55e","Type":"ContainerStarted","Data":"2ba5cdc510bd73817bb539da627a54c562e2385e6fa1c0d93d361d9f094312fe"}
Oct 31 02:15:02.715912 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:02.715890    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2" event={"ID":"e0ae7217-5d7c-4dd3-b830-7e669721ec67","Type":"ContainerStarted","Data":"7e0784e5b6598592e434a98ff6fa28a857e6012698ae470f2b56c78fff87e690"}
Oct 31 02:15:02.716016 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:02.715926    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2" event={"ID":"e0ae7217-5d7c-4dd3-b830-7e669721ec67","Type":"ContainerStarted","Data":"185f47514f57f077fa68b7b5b8d3831cc2ab832c4822ace68374c09db540739f"}
Oct 31 02:15:02.717545 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:02.717525    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs" event={"ID":"dfceee3a-9e84-4888-a7ff-bb5fe5b632e5","Type":"ContainerStarted","Data":"f38dedbf6b69d0822df652f4f802b7074d539789eb77225939a3769963143378"}
Oct 31 02:15:02.719205 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:02.719179    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx" event={"ID":"0c8cd351-2df1-4f99-a782-e63b9aab297f","Type":"ContainerStarted","Data":"f1b4b94c46953c13f3c08714a0ff0349248f314fbaaf6d11fec4904f33c021d5"}
Oct 31 02:15:02.720998 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:02.720974    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj" event={"ID":"ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf","Type":"ContainerStarted","Data":"1a04de3853c8ad15e06b6f1857bb35fbeebb04fc8d137d3869a174247f5db6f2"}
Oct 31 02:15:02.724150 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:02.724127    5650 generic.go:356] "Generic (PLEG): container finished" podID="f6566d82-f8bc-44f8-b4ff-21edc8e67810" containerID="1929b042c06462ebc5cece1e1b52ca31574f661e32daeff4789775a50285e340" exitCode=60
Oct 31 02:15:02.724228 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:02.724209    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84" event={"ID":"f6566d82-f8bc-44f8-b4ff-21edc8e67810","Type":"ContainerDied","Data":"1929b042c06462ebc5cece1e1b52ca31574f661e32daeff4789775a50285e340"}
Oct 31 02:15:02.724436 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:02.724418    5650 scope.go:122] "RemoveContainer" containerID="1929b042c06462ebc5cece1e1b52ca31574f661e32daeff4789775a50285e340"
Oct 31 02:15:02.731970 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:02.731926    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql" podStartSLOduration=2.731911572 podStartE2EDuration="2.731911572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:15:02.726972482 +0000 UTC m=+3258.730686774" watchObservedRunningTime="2026-10-31 02:15:02.731911572 +0000 UTC m=+3258.735625862"
Oct 31 02:15:02.750540 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:02.750493    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7" podStartSLOduration=2.750476881 podStartE2EDuration="2.750476881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:15:02.746612411 +0000 UTC m=+3258.750326700" watchObservedRunningTime="2026-10-31 02:15:02.750476881 +0000 UTC m=+3258.754191173"
Oct 31 02:15:02.768942 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:02.768848    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw" podStartSLOduration=2.768833332 podStartE2EDuration="2.768833332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:15:02.765025302 +0000 UTC m=+3258.768739594" watchObservedRunningTime="2026-10-31 02:15:02.768833332 +0000 UTC m=+3258.772547623"
Oct 31 02:15:02.788934 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:02.788610    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx" podStartSLOduration=2.788595175 podStartE2EDuration="2.788595175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:15:02.784255793 +0000 UTC m=+3258.787970085" watchObservedRunningTime="2026-10-31 02:15:02.788595175 +0000 UTC m=+3258.792309500"
Oct 31 02:15:02.806303 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:02.806046    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2" podStartSLOduration=2.806028545 podStartE2EDuration="2.806028545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:15:02.803878926 +0000 UTC m=+3258.807593219" watchObservedRunningTime="2026-10-31 02:15:02.806028545 +0000 UTC m=+3258.809742836"
Oct 31 02:15:02.848439 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:02.848381    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj" podStartSLOduration=2.848366974 podStartE2EDuration="2.848366974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:15:02.847834302 +0000 UTC m=+3258.851548602" watchObservedRunningTime="2026-10-31 02:15:02.848366974 +0000 UTC m=+3258.852081264"
Oct 31 02:15:02.871839 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:02.871778    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs" podStartSLOduration=2.871765233 podStartE2EDuration="2.871765233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:15:02.870582056 +0000 UTC m=+3258.874296362" watchObservedRunningTime="2026-10-31 02:15:02.871765233 +0000 UTC m=+3258.875479524"
Oct 31 02:15:02.894266 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:02.894211    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7" podStartSLOduration=2.8941957780000003 podStartE2EDuration="2.894195778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:15:02.890082144 +0000 UTC m=+3258.893796438" watchObservedRunningTime="2026-10-31 02:15:02.894195778 +0000 UTC m=+3258.897910068"
Oct 31 02:15:03.734144 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.734106    5650 generic.go:356] "Generic (PLEG): container finished" podID="6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d" containerID="f8d81bdacf514121bf2a9727ebdcc61ec958f79720dc2da78c76eabb6879ac92" exitCode=60
Oct 31 02:15:03.734144 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.734138    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw" event={"ID":"6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d","Type":"ContainerDied","Data":"f8d81bdacf514121bf2a9727ebdcc61ec958f79720dc2da78c76eabb6879ac92"}
Oct 31 02:15:03.734649 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.734560    5650 scope.go:122] "RemoveContainer" containerID="f8d81bdacf514121bf2a9727ebdcc61ec958f79720dc2da78c76eabb6879ac92"
Oct 31 02:15:03.736110 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.736083    5650 generic.go:356] "Generic (PLEG): container finished" podID="9c3ce5c1-cd13-429e-9734-916e37edfb8a" containerID="4749283eb928c88d4bfb36535c3457c745e17d9532521e7dc7e6d3f1dbb689a9" exitCode=60
Oct 31 02:15:03.736226 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.736135    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7" event={"ID":"9c3ce5c1-cd13-429e-9734-916e37edfb8a","Type":"ContainerDied","Data":"4749283eb928c88d4bfb36535c3457c745e17d9532521e7dc7e6d3f1dbb689a9"}
Oct 31 02:15:03.736535 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.736438    5650 scope.go:122] "RemoveContainer" containerID="4749283eb928c88d4bfb36535c3457c745e17d9532521e7dc7e6d3f1dbb689a9"
Oct 31 02:15:03.738168 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.738071    5650 generic.go:356] "Generic (PLEG): container finished" podID="495060eb-5f49-4f96-bfe7-b89ed0c8b55e" containerID="2ba5cdc510bd73817bb539da627a54c562e2385e6fa1c0d93d361d9f094312fe" exitCode=60
Oct 31 02:15:03.738168 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.738130    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7" event={"ID":"495060eb-5f49-4f96-bfe7-b89ed0c8b55e","Type":"ContainerDied","Data":"2ba5cdc510bd73817bb539da627a54c562e2385e6fa1c0d93d361d9f094312fe"}
Oct 31 02:15:03.738480 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.738407    5650 scope.go:122] "RemoveContainer" containerID="2ba5cdc510bd73817bb539da627a54c562e2385e6fa1c0d93d361d9f094312fe"
Oct 31 02:15:03.740271 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.740245    5650 generic.go:356] "Generic (PLEG): container finished" podID="e0ae7217-5d7c-4dd3-b830-7e669721ec67" containerID="7e0784e5b6598592e434a98ff6fa28a857e6012698ae470f2b56c78fff87e690" exitCode=60
Oct 31 02:15:03.740365 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.740320    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2" event={"ID":"e0ae7217-5d7c-4dd3-b830-7e669721ec67","Type":"ContainerDied","Data":"7e0784e5b6598592e434a98ff6fa28a857e6012698ae470f2b56c78fff87e690"}
Oct 31 02:15:03.740607 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.740591    5650 scope.go:122] "RemoveContainer" containerID="7e0784e5b6598592e434a98ff6fa28a857e6012698ae470f2b56c78fff87e690"
Oct 31 02:15:03.741988 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.741969    5650 generic.go:356] "Generic (PLEG): container finished" podID="dfceee3a-9e84-4888-a7ff-bb5fe5b632e5" containerID="f38dedbf6b69d0822df652f4f802b7074d539789eb77225939a3769963143378" exitCode=60
Oct 31 02:15:03.742696 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.742057    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs" event={"ID":"dfceee3a-9e84-4888-a7ff-bb5fe5b632e5","Type":"ContainerDied","Data":"f38dedbf6b69d0822df652f4f802b7074d539789eb77225939a3769963143378"}
Oct 31 02:15:03.742696 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.742310    5650 scope.go:122] "RemoveContainer" containerID="f38dedbf6b69d0822df652f4f802b7074d539789eb77225939a3769963143378"
Oct 31 02:15:03.743987 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.743966    5650 generic.go:356] "Generic (PLEG): container finished" podID="0c8cd351-2df1-4f99-a782-e63b9aab297f" containerID="f1b4b94c46953c13f3c08714a0ff0349248f314fbaaf6d11fec4904f33c021d5" exitCode=60
Oct 31 02:15:03.744067 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.744009    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx" event={"ID":"0c8cd351-2df1-4f99-a782-e63b9aab297f","Type":"ContainerDied","Data":"f1b4b94c46953c13f3c08714a0ff0349248f314fbaaf6d11fec4904f33c021d5"}
Oct 31 02:15:03.744274 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.744257    5650 scope.go:122] "RemoveContainer" containerID="f1b4b94c46953c13f3c08714a0ff0349248f314fbaaf6d11fec4904f33c021d5"
Oct 31 02:15:03.745595 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.745577    5650 generic.go:356] "Generic (PLEG): container finished" podID="ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf" containerID="1a04de3853c8ad15e06b6f1857bb35fbeebb04fc8d137d3869a174247f5db6f2" exitCode=60
Oct 31 02:15:03.745693 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.745674    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj" event={"ID":"ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf","Type":"ContainerDied","Data":"1a04de3853c8ad15e06b6f1857bb35fbeebb04fc8d137d3869a174247f5db6f2"}
Oct 31 02:15:03.745866 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.745848    5650 scope.go:122] "RemoveContainer" containerID="1a04de3853c8ad15e06b6f1857bb35fbeebb04fc8d137d3869a174247f5db6f2"
Oct 31 02:15:03.747809 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.747777    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84" event={"ID":"f6566d82-f8bc-44f8-b4ff-21edc8e67810","Type":"ContainerStarted","Data":"2a22edfb643274495040e1e9b9569906c3bc078bb9d554d69fa878a1d5d8e76d"}
Oct 31 02:15:03.749832 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.749810    5650 generic.go:356] "Generic (PLEG): container finished" podID="87ac2c5d-5b80-44ce-904b-713134c4b4f6" containerID="73615ec0619cc62c7a68b0c347dc55b0361a961892031d1354cc210623d2ecbe" exitCode=60
Oct 31 02:15:03.749935 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.749842    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql" event={"ID":"87ac2c5d-5b80-44ce-904b-713134c4b4f6","Type":"ContainerDied","Data":"73615ec0619cc62c7a68b0c347dc55b0361a961892031d1354cc210623d2ecbe"}
Oct 31 02:15:03.750152 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.750105    5650 scope.go:122] "RemoveContainer" containerID="73615ec0619cc62c7a68b0c347dc55b0361a961892031d1354cc210623d2ecbe"
Oct 31 02:15:03.821919 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:03.821871    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84" podStartSLOduration=3.821857045 podStartE2EDuration="3.821857045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:15:03.811350881 +0000 UTC m=+3259.815065210" watchObservedRunningTime="2026-10-31 02:15:03.821857045 +0000 UTC m=+3259.825571363"
Oct 31 02:15:04.770358 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:04.770313    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7" event={"ID":"9c3ce5c1-cd13-429e-9734-916e37edfb8a","Type":"ContainerStarted","Data":"2169185db72349323a81ea45d19bea7161f2a6dd3333a5af64f30c4b5b75c705"}
Oct 31 02:15:04.775659 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:04.775155    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7" event={"ID":"495060eb-5f49-4f96-bfe7-b89ed0c8b55e","Type":"ContainerStarted","Data":"0d80974b511d93004547299e114378c008b98f2b5d636ae954574c3c658de661"}
Oct 31 02:15:04.778025 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:04.777993    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2" event={"ID":"e0ae7217-5d7c-4dd3-b830-7e669721ec67","Type":"ContainerStarted","Data":"6621f6e31359a601f53633e17b95f341e71a536dcfd2d7d42efe53342ee417f1"}
Oct 31 02:15:04.780190 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:04.780163    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs" event={"ID":"dfceee3a-9e84-4888-a7ff-bb5fe5b632e5","Type":"ContainerStarted","Data":"be387e9b534366a46d93f99721e58ce138f3f1ecc038680a3ad8f978dd53767c"}
Oct 31 02:15:04.782522 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:04.782493    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx" event={"ID":"0c8cd351-2df1-4f99-a782-e63b9aab297f","Type":"ContainerStarted","Data":"586f1e98f062d7c104fa4202ef509bdb1af66951d4eaadfe58507da5b5c2bdee"}
Oct 31 02:15:04.784354 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:04.784332    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj" event={"ID":"ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf","Type":"ContainerStarted","Data":"c568c4f9fb037d474583693db438082b2eb89e1beff8636a738b08703e3b0df4"}
Oct 31 02:15:04.786231 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:04.786209    5650 generic.go:356] "Generic (PLEG): container finished" podID="f6566d82-f8bc-44f8-b4ff-21edc8e67810" containerID="2a22edfb643274495040e1e9b9569906c3bc078bb9d554d69fa878a1d5d8e76d" exitCode=60
Oct 31 02:15:04.786338 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:04.786286    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84" event={"ID":"f6566d82-f8bc-44f8-b4ff-21edc8e67810","Type":"ContainerDied","Data":"2a22edfb643274495040e1e9b9569906c3bc078bb9d554d69fa878a1d5d8e76d"}
Oct 31 02:15:04.786338 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:04.786333    5650 scope.go:122] "RemoveContainer" containerID="1929b042c06462ebc5cece1e1b52ca31574f661e32daeff4789775a50285e340"
Oct 31 02:15:04.786565 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:04.786539    5650 scope.go:122] "RemoveContainer" containerID="2a22edfb643274495040e1e9b9569906c3bc078bb9d554d69fa878a1d5d8e76d"
Oct 31 02:15:04.786868 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:04.786843    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84_odh-ai-gateway-infra(f6566d82-f8bc-44f8-b4ff-21edc8e67810)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84" podUID="f6566d82-f8bc-44f8-b4ff-21edc8e67810"
Oct 31 02:15:04.789193 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:04.789169    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql" event={"ID":"87ac2c5d-5b80-44ce-904b-713134c4b4f6","Type":"ContainerStarted","Data":"6d548abc02fb95b35ff50bebd79d67f76e59f8832794205a7f9c2e719f2f35fa"}
Oct 31 02:15:04.793235 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:04.792775    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw" event={"ID":"6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d","Type":"ContainerStarted","Data":"2e2bfeb3c5ee0f47950ebca331a81dbdf92817dd0238219ecfdbb48e6749a4a1"}
Oct 31 02:15:05.805484 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.805434    5650 generic.go:356] "Generic (PLEG): container finished" podID="e0ae7217-5d7c-4dd3-b830-7e669721ec67" containerID="6621f6e31359a601f53633e17b95f341e71a536dcfd2d7d42efe53342ee417f1" exitCode=60
Oct 31 02:15:05.805852 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.805495    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2" event={"ID":"e0ae7217-5d7c-4dd3-b830-7e669721ec67","Type":"ContainerDied","Data":"6621f6e31359a601f53633e17b95f341e71a536dcfd2d7d42efe53342ee417f1"}
Oct 31 02:15:05.805852 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.805599    5650 scope.go:122] "RemoveContainer" containerID="7e0784e5b6598592e434a98ff6fa28a857e6012698ae470f2b56c78fff87e690"
Oct 31 02:15:05.805985 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.805949    5650 scope.go:122] "RemoveContainer" containerID="6621f6e31359a601f53633e17b95f341e71a536dcfd2d7d42efe53342ee417f1"
Oct 31 02:15:05.806436 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:05.806406    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890215-sn5w2_odh-ai-gateway-infra(e0ae7217-5d7c-4dd3-b830-7e669721ec67)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2" podUID="e0ae7217-5d7c-4dd3-b830-7e669721ec67"
Oct 31 02:15:05.807935 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.807908    5650 generic.go:356] "Generic (PLEG): container finished" podID="dfceee3a-9e84-4888-a7ff-bb5fe5b632e5" containerID="be387e9b534366a46d93f99721e58ce138f3f1ecc038680a3ad8f978dd53767c" exitCode=60
Oct 31 02:15:05.808055 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.807984    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs" event={"ID":"dfceee3a-9e84-4888-a7ff-bb5fe5b632e5","Type":"ContainerDied","Data":"be387e9b534366a46d93f99721e58ce138f3f1ecc038680a3ad8f978dd53767c"}
Oct 31 02:15:05.808374 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.808355    5650 scope.go:122] "RemoveContainer" containerID="be387e9b534366a46d93f99721e58ce138f3f1ecc038680a3ad8f978dd53767c"
Oct 31 02:15:05.808885 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:05.808854    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs_odh-ai-gateway-infra(dfceee3a-9e84-4888-a7ff-bb5fe5b632e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs" podUID="dfceee3a-9e84-4888-a7ff-bb5fe5b632e5"
Oct 31 02:15:05.810461 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.810426    5650 generic.go:356] "Generic (PLEG): container finished" podID="0c8cd351-2df1-4f99-a782-e63b9aab297f" containerID="586f1e98f062d7c104fa4202ef509bdb1af66951d4eaadfe58507da5b5c2bdee" exitCode=60
Oct 31 02:15:05.810576 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.810496    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx" event={"ID":"0c8cd351-2df1-4f99-a782-e63b9aab297f","Type":"ContainerDied","Data":"586f1e98f062d7c104fa4202ef509bdb1af66951d4eaadfe58507da5b5c2bdee"}
Oct 31 02:15:05.811013 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.810987    5650 scope.go:122] "RemoveContainer" containerID="586f1e98f062d7c104fa4202ef509bdb1af66951d4eaadfe58507da5b5c2bdee"
Oct 31 02:15:05.811561 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:05.811533    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx_odh-ai-gateway-infra(0c8cd351-2df1-4f99-a782-e63b9aab297f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx" podUID="0c8cd351-2df1-4f99-a782-e63b9aab297f"
Oct 31 02:15:05.813386 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.813363    5650 generic.go:356] "Generic (PLEG): container finished" podID="ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf" containerID="c568c4f9fb037d474583693db438082b2eb89e1beff8636a738b08703e3b0df4" exitCode=60
Oct 31 02:15:05.813785 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.813763    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj" event={"ID":"ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf","Type":"ContainerDied","Data":"c568c4f9fb037d474583693db438082b2eb89e1beff8636a738b08703e3b0df4"}
Oct 31 02:15:05.813872 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.813791    5650 scope.go:122] "RemoveContainer" containerID="c568c4f9fb037d474583693db438082b2eb89e1beff8636a738b08703e3b0df4"
Oct 31 02:15:05.814232 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:05.814196    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj_odh-ai-gateway-infra(ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj" podUID="ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf"
Oct 31 02:15:05.816053 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.816030    5650 scope.go:122] "RemoveContainer" containerID="2a22edfb643274495040e1e9b9569906c3bc078bb9d554d69fa878a1d5d8e76d"
Oct 31 02:15:05.816441 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:05.816416    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84_odh-ai-gateway-infra(f6566d82-f8bc-44f8-b4ff-21edc8e67810)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84" podUID="f6566d82-f8bc-44f8-b4ff-21edc8e67810"
Oct 31 02:15:05.818145 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.818122    5650 generic.go:356] "Generic (PLEG): container finished" podID="87ac2c5d-5b80-44ce-904b-713134c4b4f6" containerID="6d548abc02fb95b35ff50bebd79d67f76e59f8832794205a7f9c2e719f2f35fa" exitCode=60
Oct 31 02:15:05.818557 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.818530    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql" event={"ID":"87ac2c5d-5b80-44ce-904b-713134c4b4f6","Type":"ContainerDied","Data":"6d548abc02fb95b35ff50bebd79d67f76e59f8832794205a7f9c2e719f2f35fa"}
Oct 31 02:15:05.818810 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.818793    5650 scope.go:122] "RemoveContainer" containerID="6d548abc02fb95b35ff50bebd79d67f76e59f8832794205a7f9c2e719f2f35fa"
Oct 31 02:15:05.819410 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:05.819373    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql_odh-ai-gateway-infra(87ac2c5d-5b80-44ce-904b-713134c4b4f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql" podUID="87ac2c5d-5b80-44ce-904b-713134c4b4f6"
Oct 31 02:15:05.820928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.820903    5650 generic.go:356] "Generic (PLEG): container finished" podID="6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d" containerID="2e2bfeb3c5ee0f47950ebca331a81dbdf92817dd0238219ecfdbb48e6749a4a1" exitCode=60
Oct 31 02:15:05.821538 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.821515    5650 scope.go:122] "RemoveContainer" containerID="2e2bfeb3c5ee0f47950ebca331a81dbdf92817dd0238219ecfdbb48e6749a4a1"
Oct 31 02:15:05.821639 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.821520    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw" event={"ID":"6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d","Type":"ContainerDied","Data":"2e2bfeb3c5ee0f47950ebca331a81dbdf92817dd0238219ecfdbb48e6749a4a1"}
Oct 31 02:15:05.821929 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:05.821904    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw_odh-ai-gateway-infra(6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw" podUID="6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d"
Oct 31 02:15:05.823073 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.823050    5650 scope.go:122] "RemoveContainer" containerID="f38dedbf6b69d0822df652f4f802b7074d539789eb77225939a3769963143378"
Oct 31 02:15:05.823530 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.823508    5650 generic.go:356] "Generic (PLEG): container finished" podID="9c3ce5c1-cd13-429e-9734-916e37edfb8a" containerID="2169185db72349323a81ea45d19bea7161f2a6dd3333a5af64f30c4b5b75c705" exitCode=60
Oct 31 02:15:05.823622 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.823555    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7" event={"ID":"9c3ce5c1-cd13-429e-9734-916e37edfb8a","Type":"ContainerDied","Data":"2169185db72349323a81ea45d19bea7161f2a6dd3333a5af64f30c4b5b75c705"}
Oct 31 02:15:05.823844 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.823830    5650 scope.go:122] "RemoveContainer" containerID="2169185db72349323a81ea45d19bea7161f2a6dd3333a5af64f30c4b5b75c705"
Oct 31 02:15:05.824139 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:05.824121    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7_odh-ai-gateway-infra(9c3ce5c1-cd13-429e-9734-916e37edfb8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7" podUID="9c3ce5c1-cd13-429e-9734-916e37edfb8a"
Oct 31 02:15:05.825901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.825881    5650 generic.go:356] "Generic (PLEG): container finished" podID="e0fa4248-e818-450c-bf1b-74f859787788" containerID="551e174065028f94449a6fa7da1b56f484f5e2157051a24e6a28b8e075e25d3b" exitCode=0
Oct 31 02:15:05.826009 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.825963    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr" event={"ID":"e0fa4248-e818-450c-bf1b-74f859787788","Type":"ContainerDied","Data":"551e174065028f94449a6fa7da1b56f484f5e2157051a24e6a28b8e075e25d3b"}
Oct 31 02:15:05.828596 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.828572    5650 generic.go:356] "Generic (PLEG): container finished" podID="495060eb-5f49-4f96-bfe7-b89ed0c8b55e" containerID="0d80974b511d93004547299e114378c008b98f2b5d636ae954574c3c658de661" exitCode=60
Oct 31 02:15:05.828697 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.828677    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7" event={"ID":"495060eb-5f49-4f96-bfe7-b89ed0c8b55e","Type":"ContainerDied","Data":"0d80974b511d93004547299e114378c008b98f2b5d636ae954574c3c658de661"}
Oct 31 02:15:05.829583 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.829558    5650 scope.go:122] "RemoveContainer" containerID="0d80974b511d93004547299e114378c008b98f2b5d636ae954574c3c658de661"
Oct 31 02:15:05.830115 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:05.829990    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7_odh-ai-gateway-infra(495060eb-5f49-4f96-bfe7-b89ed0c8b55e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7" podUID="495060eb-5f49-4f96-bfe7-b89ed0c8b55e"
Oct 31 02:15:05.846585 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.846555    5650 scope.go:122] "RemoveContainer" containerID="f1b4b94c46953c13f3c08714a0ff0349248f314fbaaf6d11fec4904f33c021d5"
Oct 31 02:15:05.869745 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.869719    5650 scope.go:122] "RemoveContainer" containerID="1a04de3853c8ad15e06b6f1857bb35fbeebb04fc8d137d3869a174247f5db6f2"
Oct 31 02:15:05.917778 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.916616    5650 scope.go:122] "RemoveContainer" containerID="73615ec0619cc62c7a68b0c347dc55b0361a961892031d1354cc210623d2ecbe"
Oct 31 02:15:05.944017 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.943983    5650 scope.go:122] "RemoveContainer" containerID="f8d81bdacf514121bf2a9727ebdcc61ec958f79720dc2da78c76eabb6879ac92"
Oct 31 02:15:05.968615 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:05.968583    5650 scope.go:122] "RemoveContainer" containerID="4749283eb928c88d4bfb36535c3457c745e17d9532521e7dc7e6d3f1dbb689a9"
Oct 31 02:15:06.010861 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.010837    5650 scope.go:122] "RemoveContainer" containerID="2ba5cdc510bd73817bb539da627a54c562e2385e6fa1c0d93d361d9f094312fe"
Oct 31 02:15:06.011829 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.011800    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:15:06.117372 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.117335    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0fa4248-e818-450c-bf1b-74f859787788-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0fa4248-e818-450c-bf1b-74f859787788-istio-podinfo\") pod \"e0fa4248-e818-450c-bf1b-74f859787788\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") "
Oct 31 02:15:06.117564 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.117501    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0fa4248-e818-450c-bf1b-74f859787788-kube-api-access-vrhl5\" (UniqueName: \"kubernetes.io/projected/e0fa4248-e818-450c-bf1b-74f859787788-kube-api-access-vrhl5\") pod \"e0fa4248-e818-450c-bf1b-74f859787788\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") "
Oct 31 02:15:06.117564 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.117548    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-istio-data\") pod \"e0fa4248-e818-450c-bf1b-74f859787788\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") "
Oct 31 02:15:06.117704 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.117576    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0fa4248-e818-450c-bf1b-74f859787788-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0fa4248-e818-450c-bf1b-74f859787788-istiod-ca-cert\") pod \"e0fa4248-e818-450c-bf1b-74f859787788\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") "
Oct 31 02:15:06.117704 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.117604    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-credential-socket\") pod \"e0fa4248-e818-450c-bf1b-74f859787788\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") "
Oct 31 02:15:06.117826 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.117700    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0fa4248-e818-450c-bf1b-74f859787788-istio-token\" (UniqueName: \"kubernetes.io/projected/e0fa4248-e818-450c-bf1b-74f859787788-istio-token\") pod \"e0fa4248-e818-450c-bf1b-74f859787788\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") "
Oct 31 02:15:06.117826 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.117740    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-workload-certs\") pod \"e0fa4248-e818-450c-bf1b-74f859787788\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") "
Oct 31 02:15:06.117826 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.117814    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-workload-socket\") pod \"e0fa4248-e818-450c-bf1b-74f859787788\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") "
Oct 31 02:15:06.118114 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.118095    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-istio-envoy\") pod \"e0fa4248-e818-450c-bf1b-74f859787788\" (UID: \"e0fa4248-e818-450c-bf1b-74f859787788\") "
Oct 31 02:15:06.119131 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.118546    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-credential-socket" pod "e0fa4248-e818-450c-bf1b-74f859787788" (UID: "e0fa4248-e818-450c-bf1b-74f859787788"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:15:06.119131 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.118686    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-istio-data" pod "e0fa4248-e818-450c-bf1b-74f859787788" (UID: "e0fa4248-e818-450c-bf1b-74f859787788"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:15:06.119131 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.118982    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:06.119131 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.119003    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:06.119131 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.119010    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0fa4248-e818-450c-bf1b-74f859787788-istiod-ca-cert" pod "e0fa4248-e818-450c-bf1b-74f859787788" (UID: "e0fa4248-e818-450c-bf1b-74f859787788"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:06.119131 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.119094    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-workload-certs" pod "e0fa4248-e818-450c-bf1b-74f859787788" (UID: "e0fa4248-e818-450c-bf1b-74f859787788"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:15:06.119799 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.119217    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-workload-socket" pod "e0fa4248-e818-450c-bf1b-74f859787788" (UID: "e0fa4248-e818-450c-bf1b-74f859787788"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:15:06.120142 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.120116    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0fa4248-e818-450c-bf1b-74f859787788-istio-podinfo" pod "e0fa4248-e818-450c-bf1b-74f859787788" (UID: "e0fa4248-e818-450c-bf1b-74f859787788"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:15:06.120895 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.120864    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-istio-envoy" pod "e0fa4248-e818-450c-bf1b-74f859787788" (UID: "e0fa4248-e818-450c-bf1b-74f859787788"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:15:06.120986 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.120874    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0fa4248-e818-450c-bf1b-74f859787788-istio-token" pod "e0fa4248-e818-450c-bf1b-74f859787788" (UID: "e0fa4248-e818-450c-bf1b-74f859787788"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:06.121601 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.121576    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0fa4248-e818-450c-bf1b-74f859787788-kube-api-access-vrhl5" pod "e0fa4248-e818-450c-bf1b-74f859787788" (UID: "e0fa4248-e818-450c-bf1b-74f859787788"). InnerVolumeSpecName "kube-api-access-vrhl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:06.220402 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.220353    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:06.220402 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.220393    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0fa4248-e818-450c-bf1b-74f859787788-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:06.220402 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.220409    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrhl5\" (UniqueName: \"kubernetes.io/projected/e0fa4248-e818-450c-bf1b-74f859787788-kube-api-access-vrhl5\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:06.220794 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.220419    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0fa4248-e818-450c-bf1b-74f859787788-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:06.220794 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.220429    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0fa4248-e818-450c-bf1b-74f859787788-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:06.220794 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.220438    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:06.220794 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.220515    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0fa4248-e818-450c-bf1b-74f859787788-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:06.839266 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.839238    5650 scope.go:122] "RemoveContainer" containerID="586f1e98f062d7c104fa4202ef509bdb1af66951d4eaadfe58507da5b5c2bdee"
Oct 31 02:15:06.839722 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:06.839626    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx_odh-ai-gateway-infra(0c8cd351-2df1-4f99-a782-e63b9aab297f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx" podUID="0c8cd351-2df1-4f99-a782-e63b9aab297f"
Oct 31 02:15:06.841076 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.841054    5650 scope.go:122] "RemoveContainer" containerID="c568c4f9fb037d474583693db438082b2eb89e1beff8636a738b08703e3b0df4"
Oct 31 02:15:06.841408 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:06.841384    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj_odh-ai-gateway-infra(ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj" podUID="ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf"
Oct 31 02:15:06.842763 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.842745    5650 scope.go:122] "RemoveContainer" containerID="6d548abc02fb95b35ff50bebd79d67f76e59f8832794205a7f9c2e719f2f35fa"
Oct 31 02:15:06.843030 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:06.843008    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql_odh-ai-gateway-infra(87ac2c5d-5b80-44ce-904b-713134c4b4f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql" podUID="87ac2c5d-5b80-44ce-904b-713134c4b4f6"
Oct 31 02:15:06.844417 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.844391    5650 scope.go:122] "RemoveContainer" containerID="2e2bfeb3c5ee0f47950ebca331a81dbdf92817dd0238219ecfdbb48e6749a4a1"
Oct 31 02:15:06.844742 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:06.844718    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw_odh-ai-gateway-infra(6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw" podUID="6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d"
Oct 31 02:15:06.846069 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.846049    5650 scope.go:122] "RemoveContainer" containerID="2169185db72349323a81ea45d19bea7161f2a6dd3333a5af64f30c4b5b75c705"
Oct 31 02:15:06.846337 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:06.846315    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7_odh-ai-gateway-infra(9c3ce5c1-cd13-429e-9734-916e37edfb8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7" podUID="9c3ce5c1-cd13-429e-9734-916e37edfb8a"
Oct 31 02:15:06.847178 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.847161    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"
Oct 31 02:15:06.847288 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.847196    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr" event={"ID":"e0fa4248-e818-450c-bf1b-74f859787788","Type":"ContainerDied","Data":"91a0be35cb2d3600d2cb53a8455d60dc1bc6a100463791dcce756e317c1ce838"}
Oct 31 02:15:06.847288 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.847257    5650 scope.go:122] "RemoveContainer" containerID="551e174065028f94449a6fa7da1b56f484f5e2157051a24e6a28b8e075e25d3b"
Oct 31 02:15:06.849338 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.849318    5650 scope.go:122] "RemoveContainer" containerID="0d80974b511d93004547299e114378c008b98f2b5d636ae954574c3c658de661"
Oct 31 02:15:06.849832 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:06.849595    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7_odh-ai-gateway-infra(495060eb-5f49-4f96-bfe7-b89ed0c8b55e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7" podUID="495060eb-5f49-4f96-bfe7-b89ed0c8b55e"
Oct 31 02:15:06.851367 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.851328    5650 scope.go:122] "RemoveContainer" containerID="6621f6e31359a601f53633e17b95f341e71a536dcfd2d7d42efe53342ee417f1"
Oct 31 02:15:06.851663 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:06.851636    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890215-sn5w2_odh-ai-gateway-infra(e0ae7217-5d7c-4dd3-b830-7e669721ec67)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2" podUID="e0ae7217-5d7c-4dd3-b830-7e669721ec67"
Oct 31 02:15:06.853236 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.853216    5650 scope.go:122] "RemoveContainer" containerID="be387e9b534366a46d93f99721e58ce138f3f1ecc038680a3ad8f978dd53767c"
Oct 31 02:15:06.853467 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:06.853429    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs_odh-ai-gateway-infra(dfceee3a-9e84-4888-a7ff-bb5fe5b632e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs" podUID="dfceee3a-9e84-4888-a7ff-bb5fe5b632e5"
Oct 31 02:15:06.938277 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.938242    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"]
Oct 31 02:15:06.947166 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:06.947134    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0b6abf-openshift-default-7764dc5fd5-qklmr"]
Oct 31 02:15:08.121424 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:08.121389    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0fa4248-e818-450c-bf1b-74f859787788" path="/var/lib/kubelet/pods/e0fa4248-e818-450c-bf1b-74f859787788/volumes"
Oct 31 02:15:10.305554 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:10.305513    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2" podUID="6f0840a2-aff1-4bb6-bba8-968c0ee0e277" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:15:11.069029 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:11.068991    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w"]
Oct 31 02:15:14.116843 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:14.116814    5650 scope.go:122] "RemoveContainer" containerID="2a22edfb643274495040e1e9b9569906c3bc078bb9d554d69fa878a1d5d8e76d"
Oct 31 02:15:14.926506 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:14.926464    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84" event={"ID":"f6566d82-f8bc-44f8-b4ff-21edc8e67810","Type":"ContainerStarted","Data":"c17e92701684cd489574480e72f5d4e01a48f4928b2557a689c850b1a7e9184a"}
Oct 31 02:15:15.688520 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:15.688492    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2"
Oct 31 02:15:15.730557 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:15.730520    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0840a2-aff1-4bb6-bba8-968c0ee0e277-kube-api-access-lxczw\" (UniqueName: \"kubernetes.io/projected/6f0840a2-aff1-4bb6-bba8-968c0ee0e277-kube-api-access-lxczw\") pod \"6f0840a2-aff1-4bb6-bba8-968c0ee0e277\" (UID: \"6f0840a2-aff1-4bb6-bba8-968c0ee0e277\") "
Oct 31 02:15:15.730737 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:15.730669    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f0840a2-aff1-4bb6-bba8-968c0ee0e277-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0840a2-aff1-4bb6-bba8-968c0ee0e277-plugins-config-volume\") pod \"6f0840a2-aff1-4bb6-bba8-968c0ee0e277\" (UID: \"6f0840a2-aff1-4bb6-bba8-968c0ee0e277\") "
Oct 31 02:15:15.731028 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:15.730997    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f0840a2-aff1-4bb6-bba8-968c0ee0e277-plugins-config-volume" pod "6f0840a2-aff1-4bb6-bba8-968c0ee0e277" (UID: "6f0840a2-aff1-4bb6-bba8-968c0ee0e277"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:15.731603 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:15.731580    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0840a2-aff1-4bb6-bba8-968c0ee0e277-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:15.732857 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:15.732828    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0840a2-aff1-4bb6-bba8-968c0ee0e277-kube-api-access-lxczw" pod "6f0840a2-aff1-4bb6-bba8-968c0ee0e277" (UID: "6f0840a2-aff1-4bb6-bba8-968c0ee0e277"). InnerVolumeSpecName "kube-api-access-lxczw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:15.852470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:15.841025    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxczw\" (UniqueName: \"kubernetes.io/projected/6f0840a2-aff1-4bb6-bba8-968c0ee0e277-kube-api-access-lxczw\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:15.962567 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:15.962525    5650 generic.go:356] "Generic (PLEG): container finished" podID="f6566d82-f8bc-44f8-b4ff-21edc8e67810" containerID="c17e92701684cd489574480e72f5d4e01a48f4928b2557a689c850b1a7e9184a" exitCode=60
Oct 31 02:15:15.962797 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:15.962651    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84" event={"ID":"f6566d82-f8bc-44f8-b4ff-21edc8e67810","Type":"ContainerDied","Data":"c17e92701684cd489574480e72f5d4e01a48f4928b2557a689c850b1a7e9184a"}
Oct 31 02:15:15.962797 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:15.962710    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84"]
Oct 31 02:15:15.962797 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:15.962742    5650 scope.go:122] "RemoveContainer" containerID="2a22edfb643274495040e1e9b9569906c3bc078bb9d554d69fa878a1d5d8e76d"
Oct 31 02:15:15.963081 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:15.963058    5650 scope.go:122] "RemoveContainer" containerID="c17e92701684cd489574480e72f5d4e01a48f4928b2557a689c850b1a7e9184a"
Oct 31 02:15:15.963715 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:15.963685    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84_odh-ai-gateway-infra(f6566d82-f8bc-44f8-b4ff-21edc8e67810)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84" podUID="f6566d82-f8bc-44f8-b4ff-21edc8e67810"
Oct 31 02:15:15.967283 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:15.966712    5650 generic.go:356] "Generic (PLEG): container finished" podID="6f0840a2-aff1-4bb6-bba8-968c0ee0e277" containerID="01ac5716fee527fe7ee4d5017bf280005ff763c34f1969cbc2098feed6624dfd" exitCode=0
Oct 31 02:15:15.967283 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:15.966774    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2"
Oct 31 02:15:15.967283 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:15.966806    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2" event={"ID":"6f0840a2-aff1-4bb6-bba8-968c0ee0e277","Type":"ContainerDied","Data":"01ac5716fee527fe7ee4d5017bf280005ff763c34f1969cbc2098feed6624dfd"}
Oct 31 02:15:15.967283 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:15.966858    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2" event={"ID":"6f0840a2-aff1-4bb6-bba8-968c0ee0e277","Type":"ContainerDied","Data":"52998d680ded63864282ff95ca3442cc62c38667304a5f82b7de0e695eb98ea3"}
Oct 31 02:15:15.980240 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:15.980214    5650 scope.go:122] "RemoveContainer" containerID="01ac5716fee527fe7ee4d5017bf280005ff763c34f1969cbc2098feed6624dfd"
Oct 31 02:15:15.995464 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:15.995428    5650 scope.go:122] "RemoveContainer" containerID="01ac5716fee527fe7ee4d5017bf280005ff763c34f1969cbc2098feed6624dfd"
Oct 31 02:15:15.995835 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:15.995814    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01ac5716fee527fe7ee4d5017bf280005ff763c34f1969cbc2098feed6624dfd\": container with ID starting with 01ac5716fee527fe7ee4d5017bf280005ff763c34f1969cbc2098feed6624dfd not found: ID does not exist" containerID="01ac5716fee527fe7ee4d5017bf280005ff763c34f1969cbc2098feed6624dfd"
Oct 31 02:15:15.995923 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:15.995847    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01ac5716fee527fe7ee4d5017bf280005ff763c34f1969cbc2098feed6624dfd"} err="failed to get container status \"01ac5716fee527fe7ee4d5017bf280005ff763c34f1969cbc2098feed6624dfd\": rpc error: code = NotFound desc = could not find container \"01ac5716fee527fe7ee4d5017bf280005ff763c34f1969cbc2098feed6624dfd\": container with ID starting with 01ac5716fee527fe7ee4d5017bf280005ff763c34f1969cbc2098feed6624dfd not found: ID does not exist"
Oct 31 02:15:16.042595 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:16.042515    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2"]
Oct 31 02:15:16.070921 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:16.062475    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0b6abf-688d6d677f-99mp2"]
Oct 31 02:15:16.088502 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:16.081548    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w" podUID="55d0de06-2458-43fb-bf56-f02492a67dea" containerName="payload-pre-processing" containerID="cri-o://19bc8fb5dd527d4ffe3b73062983a8a86032c5e5a371aa6b77f2d05e582d6fe5" gracePeriod=25
Oct 31 02:15:16.128310 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:16.128271    5650 scope.go:122] "RemoveContainer" containerID="be387e9b534366a46d93f99721e58ce138f3f1ecc038680a3ad8f978dd53767c"
Oct 31 02:15:16.133134 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:16.133103    5650 scope.go:122] "RemoveContainer" containerID="2e2bfeb3c5ee0f47950ebca331a81dbdf92817dd0238219ecfdbb48e6749a4a1"
Oct 31 02:15:16.175772 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:16.175547    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f0840a2-aff1-4bb6-bba8-968c0ee0e277" path="/var/lib/kubelet/pods/6f0840a2-aff1-4bb6-bba8-968c0ee0e277/volumes"
Oct 31 02:15:16.644667 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:16.644633    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w"
Oct 31 02:15:16.680753 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:16.680713    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55d0de06-2458-43fb-bf56-f02492a67dea-kube-api-access-q8psp\" (UniqueName: \"kubernetes.io/projected/55d0de06-2458-43fb-bf56-f02492a67dea-kube-api-access-q8psp\") pod \"55d0de06-2458-43fb-bf56-f02492a67dea\" (UID: \"55d0de06-2458-43fb-bf56-f02492a67dea\") "
Oct 31 02:15:16.681064 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:16.681038    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55d0de06-2458-43fb-bf56-f02492a67dea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55d0de06-2458-43fb-bf56-f02492a67dea-plugins-config-volume\") pod \"55d0de06-2458-43fb-bf56-f02492a67dea\" (UID: \"55d0de06-2458-43fb-bf56-f02492a67dea\") "
Oct 31 02:15:16.681598 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:16.681551    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55d0de06-2458-43fb-bf56-f02492a67dea-plugins-config-volume" pod "55d0de06-2458-43fb-bf56-f02492a67dea" (UID: "55d0de06-2458-43fb-bf56-f02492a67dea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:16.681990 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:16.681965    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55d0de06-2458-43fb-bf56-f02492a67dea-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:16.683196 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:16.683169    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d0de06-2458-43fb-bf56-f02492a67dea-kube-api-access-q8psp" pod "55d0de06-2458-43fb-bf56-f02492a67dea" (UID: "55d0de06-2458-43fb-bf56-f02492a67dea"). InnerVolumeSpecName "kube-api-access-q8psp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:16.790413 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:16.785032    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8psp\" (UniqueName: \"kubernetes.io/projected/55d0de06-2458-43fb-bf56-f02492a67dea-kube-api-access-q8psp\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:16.980624 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:16.980590    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw" event={"ID":"6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d","Type":"ContainerStarted","Data":"7a45badd009e825a1a8c0e233976b8b81e24ba028df9aa3a33c8892347c66100"}
Oct 31 02:15:16.982388 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:16.982364    5650 generic.go:356] "Generic (PLEG): container finished" podID="55d0de06-2458-43fb-bf56-f02492a67dea" containerID="19bc8fb5dd527d4ffe3b73062983a8a86032c5e5a371aa6b77f2d05e582d6fe5" exitCode=0
Oct 31 02:15:16.982572 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:16.982558    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w" event={"ID":"55d0de06-2458-43fb-bf56-f02492a67dea","Type":"ContainerDied","Data":"19bc8fb5dd527d4ffe3b73062983a8a86032c5e5a371aa6b77f2d05e582d6fe5"}
Oct 31 02:15:16.982628 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:16.982594    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w" event={"ID":"55d0de06-2458-43fb-bf56-f02492a67dea","Type":"ContainerDied","Data":"b9e7ce6bb425952c040bf9c67936649c67356ea748b99b24a95fca7eb5504810"}
Oct 31 02:15:16.982628 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:16.982615    5650 scope.go:122] "RemoveContainer" containerID="19bc8fb5dd527d4ffe3b73062983a8a86032c5e5a371aa6b77f2d05e582d6fe5"
Oct 31 02:15:16.982705 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:16.982684    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w"
Oct 31 02:15:16.985294 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:16.985268    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs" event={"ID":"dfceee3a-9e84-4888-a7ff-bb5fe5b632e5","Type":"ContainerStarted","Data":"586ddf56b84a5e0447d7940e49fcb34b1f9651081c774fb45d0292975bcccc97"}
Oct 31 02:15:16.996194 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:16.996164    5650 scope.go:122] "RemoveContainer" containerID="19bc8fb5dd527d4ffe3b73062983a8a86032c5e5a371aa6b77f2d05e582d6fe5"
Oct 31 02:15:16.996595 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:16.996571    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19bc8fb5dd527d4ffe3b73062983a8a86032c5e5a371aa6b77f2d05e582d6fe5\": container with ID starting with 19bc8fb5dd527d4ffe3b73062983a8a86032c5e5a371aa6b77f2d05e582d6fe5 not found: ID does not exist" containerID="19bc8fb5dd527d4ffe3b73062983a8a86032c5e5a371aa6b77f2d05e582d6fe5"
Oct 31 02:15:16.996704 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:16.996608    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19bc8fb5dd527d4ffe3b73062983a8a86032c5e5a371aa6b77f2d05e582d6fe5"} err="failed to get container status \"19bc8fb5dd527d4ffe3b73062983a8a86032c5e5a371aa6b77f2d05e582d6fe5\": rpc error: code = NotFound desc = could not find container \"19bc8fb5dd527d4ffe3b73062983a8a86032c5e5a371aa6b77f2d05e582d6fe5\": container with ID starting with 19bc8fb5dd527d4ffe3b73062983a8a86032c5e5a371aa6b77f2d05e582d6fe5 not found: ID does not exist"
Oct 31 02:15:17.049254 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:17.047943    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w"]
Oct 31 02:15:17.054869 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:17.053760    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w"]
Oct 31 02:15:17.111485 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:17.111457    5650 scope.go:122] "RemoveContainer" containerID="586f1e98f062d7c104fa4202ef509bdb1af66951d4eaadfe58507da5b5c2bdee"
Oct 31 02:15:17.212043 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:17.212018    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84"
Oct 31 02:15:17.292803 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:17.292769    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6566d82-f8bc-44f8-b4ff-21edc8e67810-kube-api-access-xq2dc\" (UniqueName: \"kubernetes.io/projected/f6566d82-f8bc-44f8-b4ff-21edc8e67810-kube-api-access-xq2dc\") pod \"f6566d82-f8bc-44f8-b4ff-21edc8e67810\" (UID: \"f6566d82-f8bc-44f8-b4ff-21edc8e67810\") "
Oct 31 02:15:17.292989 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:17.292851    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6566d82-f8bc-44f8-b4ff-21edc8e67810-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6566d82-f8bc-44f8-b4ff-21edc8e67810-maas-api-service-ca\") pod \"f6566d82-f8bc-44f8-b4ff-21edc8e67810\" (UID: \"f6566d82-f8bc-44f8-b4ff-21edc8e67810\") "
Oct 31 02:15:17.293319 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:17.293293    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6566d82-f8bc-44f8-b4ff-21edc8e67810-maas-api-service-ca" pod "f6566d82-f8bc-44f8-b4ff-21edc8e67810" (UID: "f6566d82-f8bc-44f8-b4ff-21edc8e67810"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:17.293503 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:17.293470    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6566d82-f8bc-44f8-b4ff-21edc8e67810-maas-api-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:17.300649 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:17.300579    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6566d82-f8bc-44f8-b4ff-21edc8e67810-kube-api-access-xq2dc" pod "f6566d82-f8bc-44f8-b4ff-21edc8e67810" (UID: "f6566d82-f8bc-44f8-b4ff-21edc8e67810"). InnerVolumeSpecName "kube-api-access-xq2dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:17.394894 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:17.394856    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq2dc\" (UniqueName: \"kubernetes.io/projected/f6566d82-f8bc-44f8-b4ff-21edc8e67810-kube-api-access-xq2dc\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:17.547517 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:17.547412    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6d7f5d-6fcb6c95f5-7cg9w" podUID="55d0de06-2458-43fb-bf56-f02492a67dea" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.166:50052\" within 1s: context deadline exceeded"
Oct 31 02:15:17.547693 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:15:17.547523    5650 logging.go:55] [core] [Channel #9711 SubChannel #9712]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.166:50052", ServerName: "10.134.0.166:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.166:50052: operation was canceled"
Oct 31 02:15:17.996006 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:17.995961    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx" event={"ID":"0c8cd351-2df1-4f99-a782-e63b9aab297f","Type":"ContainerStarted","Data":"1096bf8698da5994f9574c67f6b917cfcbf046441395e7d20900a41b0bedb1c9"}
Oct 31 02:15:17.997583 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:17.997562    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84"
Oct 31 02:15:17.997714 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:17.997611    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84" event={"ID":"f6566d82-f8bc-44f8-b4ff-21edc8e67810","Type":"ContainerDied","Data":"aaffb4716719b62f0c86f6d8045f8ccd88cb0d23361647383ee8001f525984a8"}
Oct 31 02:15:17.997714 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:17.997672    5650 scope.go:122] "RemoveContainer" containerID="c17e92701684cd489574480e72f5d4e01a48f4928b2557a689c850b1a7e9184a"
Oct 31 02:15:18.000127 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:17.999967    5650 generic.go:356] "Generic (PLEG): container finished" podID="6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d" containerID="7a45badd009e825a1a8c0e233976b8b81e24ba028df9aa3a33c8892347c66100" exitCode=60
Oct 31 02:15:18.000127 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:18.000026    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw" event={"ID":"6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d","Type":"ContainerDied","Data":"7a45badd009e825a1a8c0e233976b8b81e24ba028df9aa3a33c8892347c66100"}
Oct 31 02:15:18.000628 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:18.000520    5650 scope.go:122] "RemoveContainer" containerID="7a45badd009e825a1a8c0e233976b8b81e24ba028df9aa3a33c8892347c66100"
Oct 31 02:15:18.001219 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:18.001189    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw_odh-ai-gateway-infra(6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw" podUID="6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d"
Oct 31 02:15:18.006971 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:18.006944    5650 generic.go:356] "Generic (PLEG): container finished" podID="dfceee3a-9e84-4888-a7ff-bb5fe5b632e5" containerID="586ddf56b84a5e0447d7940e49fcb34b1f9651081c774fb45d0292975bcccc97" exitCode=60
Oct 31 02:15:18.007098 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:18.006980    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs" event={"ID":"dfceee3a-9e84-4888-a7ff-bb5fe5b632e5","Type":"ContainerDied","Data":"586ddf56b84a5e0447d7940e49fcb34b1f9651081c774fb45d0292975bcccc97"}
Oct 31 02:15:18.007512 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:18.007378    5650 scope.go:122] "RemoveContainer" containerID="586ddf56b84a5e0447d7940e49fcb34b1f9651081c774fb45d0292975bcccc97"
Oct 31 02:15:18.008199 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:18.008013    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs_odh-ai-gateway-infra(dfceee3a-9e84-4888-a7ff-bb5fe5b632e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs" podUID="dfceee3a-9e84-4888-a7ff-bb5fe5b632e5"
Oct 31 02:15:18.012160 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:18.012135    5650 scope.go:122] "RemoveContainer" containerID="2e2bfeb3c5ee0f47950ebca331a81dbdf92817dd0238219ecfdbb48e6749a4a1"
Oct 31 02:15:18.028038 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:18.028008    5650 scope.go:122] "RemoveContainer" containerID="be387e9b534366a46d93f99721e58ce138f3f1ecc038680a3ad8f978dd53767c"
Oct 31 02:15:18.037051 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:18.037016    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs"]
Oct 31 02:15:18.045351 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:18.045323    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84"]
Oct 31 02:15:18.051244 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:18.051218    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-91b6b1-29890215-ltm84"]
Oct 31 02:15:18.096125 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:18.096088    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw"]
Oct 31 02:15:18.111894 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:18.111869    5650 scope.go:122] "RemoveContainer" containerID="2169185db72349323a81ea45d19bea7161f2a6dd3333a5af64f30c4b5b75c705"
Oct 31 02:15:18.112084 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:18.112050    5650 scope.go:122] "RemoveContainer" containerID="c568c4f9fb037d474583693db438082b2eb89e1beff8636a738b08703e3b0df4"
Oct 31 02:15:18.122028 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:18.122001    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55d0de06-2458-43fb-bf56-f02492a67dea" path="/var/lib/kubelet/pods/55d0de06-2458-43fb-bf56-f02492a67dea/volumes"
Oct 31 02:15:18.122471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:18.122441    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6566d82-f8bc-44f8-b4ff-21edc8e67810" path="/var/lib/kubelet/pods/f6566d82-f8bc-44f8-b4ff-21edc8e67810/volumes"
Oct 31 02:15:19.019529 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.019492    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7" event={"ID":"9c3ce5c1-cd13-429e-9734-916e37edfb8a","Type":"ContainerStarted","Data":"e4eac19078f55c8ed31c9319d1b8a2577039dd7d7034497b9812e84aba38fcb7"}
Oct 31 02:15:19.023385 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.023356    5650 generic.go:356] "Generic (PLEG): container finished" podID="0c8cd351-2df1-4f99-a782-e63b9aab297f" containerID="1096bf8698da5994f9574c67f6b917cfcbf046441395e7d20900a41b0bedb1c9" exitCode=60
Oct 31 02:15:19.023549 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.023431    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx" event={"ID":"0c8cd351-2df1-4f99-a782-e63b9aab297f","Type":"ContainerDied","Data":"1096bf8698da5994f9574c67f6b917cfcbf046441395e7d20900a41b0bedb1c9"}
Oct 31 02:15:19.023549 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.023516    5650 scope.go:122] "RemoveContainer" containerID="586f1e98f062d7c104fa4202ef509bdb1af66951d4eaadfe58507da5b5c2bdee"
Oct 31 02:15:19.023851 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.023834    5650 scope.go:122] "RemoveContainer" containerID="1096bf8698da5994f9574c67f6b917cfcbf046441395e7d20900a41b0bedb1c9"
Oct 31 02:15:19.024177 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:19.024154    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx_odh-ai-gateway-infra(0c8cd351-2df1-4f99-a782-e63b9aab297f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx" podUID="0c8cd351-2df1-4f99-a782-e63b9aab297f"
Oct 31 02:15:19.026513 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.026492    5650 generic.go:356] "Generic (PLEG): container finished" podID="ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf" containerID="845bf3eb331cb6ea69f4d7da248669253cf4edc07d3e649b75f4a042120bfae0" exitCode=60
Oct 31 02:15:19.026624 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.026564    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj" event={"ID":"ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf","Type":"ContainerDied","Data":"845bf3eb331cb6ea69f4d7da248669253cf4edc07d3e649b75f4a042120bfae0"}
Oct 31 02:15:19.026958 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.026935    5650 scope.go:122] "RemoveContainer" containerID="845bf3eb331cb6ea69f4d7da248669253cf4edc07d3e649b75f4a042120bfae0"
Oct 31 02:15:19.027319 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:19.027297    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj_odh-ai-gateway-infra(ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj" podUID="ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf"
Oct 31 02:15:19.034884 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.034860    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx"]
Oct 31 02:15:19.059390 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.059362    5650 scope.go:122] "RemoveContainer" containerID="c568c4f9fb037d474583693db438082b2eb89e1beff8636a738b08703e3b0df4"
Oct 31 02:15:19.122095 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.114009    5650 scope.go:122] "RemoveContainer" containerID="0d80974b511d93004547299e114378c008b98f2b5d636ae954574c3c658de661"
Oct 31 02:15:19.122095 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.114370    5650 scope.go:122] "RemoveContainer" containerID="6621f6e31359a601f53633e17b95f341e71a536dcfd2d7d42efe53342ee417f1"
Oct 31 02:15:19.122095 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.114959    5650 scope.go:122] "RemoveContainer" containerID="6d548abc02fb95b35ff50bebd79d67f76e59f8832794205a7f9c2e719f2f35fa"
Oct 31 02:15:19.332381 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.332348    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs"
Oct 31 02:15:19.399558 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.399525    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw"
Oct 31 02:15:19.428309 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.428228    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfceee3a-9e84-4888-a7ff-bb5fe5b632e5-kube-api-access-sgfq9\" (UniqueName: \"kubernetes.io/projected/dfceee3a-9e84-4888-a7ff-bb5fe5b632e5-kube-api-access-sgfq9\") pod \"dfceee3a-9e84-4888-a7ff-bb5fe5b632e5\" (UID: \"dfceee3a-9e84-4888-a7ff-bb5fe5b632e5\") "
Oct 31 02:15:19.428499 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.428335    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d-kube-api-access-vc9hv\" (UniqueName: \"kubernetes.io/projected/6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d-kube-api-access-vc9hv\") pod \"6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d\" (UID: \"6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d\") "
Oct 31 02:15:19.428499 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.428392    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d-maas-api-service-ca\") pod \"6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d\" (UID: \"6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d\") "
Oct 31 02:15:19.428581 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.428556    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfceee3a-9e84-4888-a7ff-bb5fe5b632e5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfceee3a-9e84-4888-a7ff-bb5fe5b632e5-maas-api-service-ca\") pod \"dfceee3a-9e84-4888-a7ff-bb5fe5b632e5\" (UID: \"dfceee3a-9e84-4888-a7ff-bb5fe5b632e5\") "
Oct 31 02:15:19.436530 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.430470    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfceee3a-9e84-4888-a7ff-bb5fe5b632e5-maas-api-service-ca" pod "dfceee3a-9e84-4888-a7ff-bb5fe5b632e5" (UID: "dfceee3a-9e84-4888-a7ff-bb5fe5b632e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:19.436530 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.430898    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d-maas-api-service-ca" pod "6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d" (UID: "6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:19.504404 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.504362    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d-kube-api-access-vc9hv" pod "6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d" (UID: "6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d"). InnerVolumeSpecName "kube-api-access-vc9hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:19.504404 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.504393    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfceee3a-9e84-4888-a7ff-bb5fe5b632e5-kube-api-access-sgfq9" pod "dfceee3a-9e84-4888-a7ff-bb5fe5b632e5" (UID: "dfceee3a-9e84-4888-a7ff-bb5fe5b632e5"). InnerVolumeSpecName "kube-api-access-sgfq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:19.566971 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.536880    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgfq9\" (UniqueName: \"kubernetes.io/projected/dfceee3a-9e84-4888-a7ff-bb5fe5b632e5-kube-api-access-sgfq9\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:19.566971 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.536904    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc9hv\" (UniqueName: \"kubernetes.io/projected/6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d-kube-api-access-vc9hv\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:19.566971 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.536913    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d-maas-api-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:19.566971 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:19.536925    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfceee3a-9e84-4888-a7ff-bb5fe5b632e5-maas-api-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:20.109752 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:20.109716    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7"]
Oct 31 02:15:20.160012 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:20.159863    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw"
Oct 31 02:15:20.165286 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:20.165248    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw" event={"ID":"6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d","Type":"ContainerDied","Data":"adaad9c8a50572651061a7cbf5d2f7523709a3a9222a1417c1cb28d9da2b72a6"}
Oct 31 02:15:20.165518 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:20.165502    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj"]
Oct 31 02:15:20.166347 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:20.166325    5650 scope.go:122] "RemoveContainer" containerID="7a45badd009e825a1a8c0e233976b8b81e24ba028df9aa3a33c8892347c66100"
Oct 31 02:15:20.171051 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:20.171028    5650 generic.go:356] "Generic (PLEG): container finished" podID="9c3ce5c1-cd13-429e-9734-916e37edfb8a" containerID="e4eac19078f55c8ed31c9319d1b8a2577039dd7d7034497b9812e84aba38fcb7" exitCode=60
Oct 31 02:15:20.171601 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:20.171234    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7" event={"ID":"9c3ce5c1-cd13-429e-9734-916e37edfb8a","Type":"ContainerDied","Data":"e4eac19078f55c8ed31c9319d1b8a2577039dd7d7034497b9812e84aba38fcb7"}
Oct 31 02:15:20.177074 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:20.177041    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7" event={"ID":"495060eb-5f49-4f96-bfe7-b89ed0c8b55e","Type":"ContainerStarted","Data":"a088324e52ed68a1ce6d7a21c5171a116e9ddde44473ccd8ec896696f9aaa21c"}
Oct 31 02:15:20.184707 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:20.184681    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2" event={"ID":"e0ae7217-5d7c-4dd3-b830-7e669721ec67","Type":"ContainerStarted","Data":"95bdd1381f6233480682691ac10f8ecfdbcb0b6123efb245a6a5b599d8ab0611"}
Oct 31 02:15:20.191288 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:20.191262    5650 scope.go:122] "RemoveContainer" containerID="2169185db72349323a81ea45d19bea7161f2a6dd3333a5af64f30c4b5b75c705"
Oct 31 02:15:20.192840 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:20.192818    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs" event={"ID":"dfceee3a-9e84-4888-a7ff-bb5fe5b632e5","Type":"ContainerDied","Data":"30cb32fddb7409b44bbae788d613d9a45b0444e79be33b904df14f6f298013a5"}
Oct 31 02:15:20.192920 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:20.192909    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs"
Oct 31 02:15:20.222406 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:20.222382    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql" event={"ID":"87ac2c5d-5b80-44ce-904b-713134c4b4f6","Type":"ContainerStarted","Data":"7bd65df32d6904f333653d660f6046b9ac674f25ee8caff55f76c110e31c02d2"}
Oct 31 02:15:20.336483 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:20.326089    5650 scope.go:122] "RemoveContainer" containerID="586ddf56b84a5e0447d7940e49fcb34b1f9651081c774fb45d0292975bcccc97"
Oct 31 02:15:20.710349 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:20.707892    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs"]
Oct 31 02:15:20.715034 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:20.714891    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d0c6ff-29890215-l25rs"]
Oct 31 02:15:20.721187 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:20.721155    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw"]
Oct 31 02:15:20.729531 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:20.728518    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5555cb-29890215-qqdpw"]
Oct 31 02:15:20.872611 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:20.872581    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj"
Oct 31 02:15:20.905760 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:20.905731    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx"
Oct 31 02:15:20.908520 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:20.908425    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7"
Oct 31 02:15:20.996470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:20.986294    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf-kube-api-access-9xw6t\" (UniqueName: \"kubernetes.io/projected/ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf-kube-api-access-9xw6t\") pod \"ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf\" (UID: \"ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf\") "
Oct 31 02:15:20.996470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:20.986871    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c8cd351-2df1-4f99-a782-e63b9aab297f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8cd351-2df1-4f99-a782-e63b9aab297f-maas-api-service-ca\") pod \"0c8cd351-2df1-4f99-a782-e63b9aab297f\" (UID: \"0c8cd351-2df1-4f99-a782-e63b9aab297f\") "
Oct 31 02:15:20.996470 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:20.991708    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c8cd351-2df1-4f99-a782-e63b9aab297f-maas-api-service-ca" pod "0c8cd351-2df1-4f99-a782-e63b9aab297f" (UID: "0c8cd351-2df1-4f99-a782-e63b9aab297f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:21.012477 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.004649    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c8cd351-2df1-4f99-a782-e63b9aab297f-kube-api-access-rz4jm\" (UniqueName: \"kubernetes.io/projected/0c8cd351-2df1-4f99-a782-e63b9aab297f-kube-api-access-rz4jm\") pod \"0c8cd351-2df1-4f99-a782-e63b9aab297f\" (UID: \"0c8cd351-2df1-4f99-a782-e63b9aab297f\") "
Oct 31 02:15:21.012477 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.004717    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf-maas-api-service-ca\") pod \"ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf\" (UID: \"ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf\") "
Oct 31 02:15:21.012477 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.005407    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c3ce5c1-cd13-429e-9734-916e37edfb8a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c3ce5c1-cd13-429e-9734-916e37edfb8a-maas-api-service-ca\") pod \"9c3ce5c1-cd13-429e-9734-916e37edfb8a\" (UID: \"9c3ce5c1-cd13-429e-9734-916e37edfb8a\") "
Oct 31 02:15:21.012477 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.005568    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c3ce5c1-cd13-429e-9734-916e37edfb8a-kube-api-access-sctpt\" (UniqueName: \"kubernetes.io/projected/9c3ce5c1-cd13-429e-9734-916e37edfb8a-kube-api-access-sctpt\") pod \"9c3ce5c1-cd13-429e-9734-916e37edfb8a\" (UID: \"9c3ce5c1-cd13-429e-9734-916e37edfb8a\") "
Oct 31 02:15:21.012477 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.007324    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c8cd351-2df1-4f99-a782-e63b9aab297f-kube-api-access-rz4jm" pod "0c8cd351-2df1-4f99-a782-e63b9aab297f" (UID: "0c8cd351-2df1-4f99-a782-e63b9aab297f"). InnerVolumeSpecName "kube-api-access-rz4jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:21.012477 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.007671    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c3ce5c1-cd13-429e-9734-916e37edfb8a-maas-api-service-ca" pod "9c3ce5c1-cd13-429e-9734-916e37edfb8a" (UID: "9c3ce5c1-cd13-429e-9734-916e37edfb8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:21.012477 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.008532    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf-maas-api-service-ca" pod "ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf" (UID: "ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:21.012477 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.008787    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8cd351-2df1-4f99-a782-e63b9aab297f-maas-api-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:21.012477 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.008807    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz4jm\" (UniqueName: \"kubernetes.io/projected/0c8cd351-2df1-4f99-a782-e63b9aab297f-kube-api-access-rz4jm\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:21.012477 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.008823    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf-maas-api-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:21.012477 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.008838    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c3ce5c1-cd13-429e-9734-916e37edfb8a-maas-api-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:21.012477 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.010957    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf-kube-api-access-9xw6t" pod "ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf" (UID: "ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf"). InnerVolumeSpecName "kube-api-access-9xw6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:21.012477 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.011750    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3ce5c1-cd13-429e-9734-916e37edfb8a-kube-api-access-sctpt" pod "9c3ce5c1-cd13-429e-9734-916e37edfb8a" (UID: "9c3ce5c1-cd13-429e-9734-916e37edfb8a"). InnerVolumeSpecName "kube-api-access-sctpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:21.110305 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.110165    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sctpt\" (UniqueName: \"kubernetes.io/projected/9c3ce5c1-cd13-429e-9734-916e37edfb8a-kube-api-access-sctpt\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:21.110305 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.110204    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xw6t\" (UniqueName: \"kubernetes.io/projected/ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf-kube-api-access-9xw6t\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:21.233652 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.233620    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx" event={"ID":"0c8cd351-2df1-4f99-a782-e63b9aab297f","Type":"ContainerDied","Data":"fda7d6a3bbe36c6dc454829e443a8c58e8e291078845f7a0dd797dcd8c835486"}
Oct 31 02:15:21.233652 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.233646    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx"
Oct 31 02:15:21.233913 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.233688    5650 scope.go:122] "RemoveContainer" containerID="1096bf8698da5994f9574c67f6b917cfcbf046441395e7d20900a41b0bedb1c9"
Oct 31 02:15:21.235334 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.235312    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj"
Oct 31 02:15:21.235475 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.235336    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj" event={"ID":"ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf","Type":"ContainerDied","Data":"114d8dfba4834c21966035728ba7ec04c8e2aa37fe143ee8466d91d910782f9d"}
Oct 31 02:15:21.237957 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.237927    5650 generic.go:356] "Generic (PLEG): container finished" podID="87ac2c5d-5b80-44ce-904b-713134c4b4f6" containerID="7bd65df32d6904f333653d660f6046b9ac674f25ee8caff55f76c110e31c02d2" exitCode=60
Oct 31 02:15:21.238083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.238053    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql" event={"ID":"87ac2c5d-5b80-44ce-904b-713134c4b4f6","Type":"ContainerDied","Data":"7bd65df32d6904f333653d660f6046b9ac674f25ee8caff55f76c110e31c02d2"}
Oct 31 02:15:21.238859 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.238819    5650 scope.go:122] "RemoveContainer" containerID="7bd65df32d6904f333653d660f6046b9ac674f25ee8caff55f76c110e31c02d2"
Oct 31 02:15:21.239231 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:21.239209    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql_odh-ai-gateway-infra(87ac2c5d-5b80-44ce-904b-713134c4b4f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql" podUID="87ac2c5d-5b80-44ce-904b-713134c4b4f6"
Oct 31 02:15:21.246104 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.245302    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7" event={"ID":"9c3ce5c1-cd13-429e-9734-916e37edfb8a","Type":"ContainerDied","Data":"61277df181baba9c7f6b03ecaabf76e4d1b1cac1fb22d2a384b3f75d62da59d2"}
Oct 31 02:15:21.246104 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.245393    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7"
Oct 31 02:15:21.246879 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.246854    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7"]
Oct 31 02:15:21.249652 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.249628    5650 generic.go:356] "Generic (PLEG): container finished" podID="495060eb-5f49-4f96-bfe7-b89ed0c8b55e" containerID="a088324e52ed68a1ce6d7a21c5171a116e9ddde44473ccd8ec896696f9aaa21c" exitCode=60
Oct 31 02:15:21.249887 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.249788    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7" event={"ID":"495060eb-5f49-4f96-bfe7-b89ed0c8b55e","Type":"ContainerDied","Data":"a088324e52ed68a1ce6d7a21c5171a116e9ddde44473ccd8ec896696f9aaa21c"}
Oct 31 02:15:21.250818 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.250653    5650 scope.go:122] "RemoveContainer" containerID="845bf3eb331cb6ea69f4d7da248669253cf4edc07d3e649b75f4a042120bfae0"
Oct 31 02:15:21.253277 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.253251    5650 generic.go:356] "Generic (PLEG): container finished" podID="e0ae7217-5d7c-4dd3-b830-7e669721ec67" containerID="95bdd1381f6233480682691ac10f8ecfdbcb0b6123efb245a6a5b599d8ab0611" exitCode=60
Oct 31 02:15:21.253430 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.253320    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2" event={"ID":"e0ae7217-5d7c-4dd3-b830-7e669721ec67","Type":"ContainerDied","Data":"95bdd1381f6233480682691ac10f8ecfdbcb0b6123efb245a6a5b599d8ab0611"}
Oct 31 02:15:21.253827 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.253804    5650 scope.go:122] "RemoveContainer" containerID="95bdd1381f6233480682691ac10f8ecfdbcb0b6123efb245a6a5b599d8ab0611"
Oct 31 02:15:21.254233 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:21.254207    5650 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890215-sn5w2_odh-ai-gateway-infra(e0ae7217-5d7c-4dd3-b830-7e669721ec67)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2" podUID="e0ae7217-5d7c-4dd3-b830-7e669721ec67"
Oct 31 02:15:21.264434 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.264407    5650 scope.go:122] "RemoveContainer" containerID="6d548abc02fb95b35ff50bebd79d67f76e59f8832794205a7f9c2e719f2f35fa"
Oct 31 02:15:21.273333 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.273284    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2"]
Oct 31 02:15:21.284026 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.284005    5650 scope.go:122] "RemoveContainer" containerID="e4eac19078f55c8ed31c9319d1b8a2577039dd7d7034497b9812e84aba38fcb7"
Oct 31 02:15:21.291718 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.291683    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7"]
Oct 31 02:15:21.301501 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.301470    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-28e4f3-29890215-csll7"]
Oct 31 02:15:21.304069 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.304044    5650 scope.go:122] "RemoveContainer" containerID="0d80974b511d93004547299e114378c008b98f2b5d636ae954574c3c658de661"
Oct 31 02:15:21.319762 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.319731    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx"]
Oct 31 02:15:21.321359 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.321332    5650 scope.go:122] "RemoveContainer" containerID="6621f6e31359a601f53633e17b95f341e71a536dcfd2d7d42efe53342ee417f1"
Oct 31 02:15:21.326239 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.326211    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2e6bfd-29890215-9clbx"]
Oct 31 02:15:21.362013 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.361932    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj"]
Oct 31 02:15:21.369007 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.368976    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed4dce-29890215-lqjvj"]
Oct 31 02:15:21.419727 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.419697    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7"
Oct 31 02:15:21.517681 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.517643    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/495060eb-5f49-4f96-bfe7-b89ed0c8b55e-kube-api-access-5wb44\" (UniqueName: \"kubernetes.io/projected/495060eb-5f49-4f96-bfe7-b89ed0c8b55e-kube-api-access-5wb44\") pod \"495060eb-5f49-4f96-bfe7-b89ed0c8b55e\" (UID: \"495060eb-5f49-4f96-bfe7-b89ed0c8b55e\") "
Oct 31 02:15:21.517901 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.517795    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/495060eb-5f49-4f96-bfe7-b89ed0c8b55e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/495060eb-5f49-4f96-bfe7-b89ed0c8b55e-maas-api-service-ca\") pod \"495060eb-5f49-4f96-bfe7-b89ed0c8b55e\" (UID: \"495060eb-5f49-4f96-bfe7-b89ed0c8b55e\") "
Oct 31 02:15:21.519055 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.519027    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/495060eb-5f49-4f96-bfe7-b89ed0c8b55e-maas-api-service-ca" pod "495060eb-5f49-4f96-bfe7-b89ed0c8b55e" (UID: "495060eb-5f49-4f96-bfe7-b89ed0c8b55e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:21.520296 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.520269    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql"]
Oct 31 02:15:21.521623 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.521598    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495060eb-5f49-4f96-bfe7-b89ed0c8b55e-kube-api-access-5wb44" pod "495060eb-5f49-4f96-bfe7-b89ed0c8b55e" (UID: "495060eb-5f49-4f96-bfe7-b89ed0c8b55e"). InnerVolumeSpecName "kube-api-access-5wb44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:21.628953 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.628858    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wb44\" (UniqueName: \"kubernetes.io/projected/495060eb-5f49-4f96-bfe7-b89ed0c8b55e-kube-api-access-5wb44\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:21.629219 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:21.629188    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/495060eb-5f49-4f96-bfe7-b89ed0c8b55e-maas-api-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:22.127243 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:22.127207    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c8cd351-2df1-4f99-a782-e63b9aab297f" path="/var/lib/kubelet/pods/0c8cd351-2df1-4f99-a782-e63b9aab297f/volumes"
Oct 31 02:15:22.127833 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:22.127811    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d" path="/var/lib/kubelet/pods/6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d/volumes"
Oct 31 02:15:22.128529 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:22.128502    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c3ce5c1-cd13-429e-9734-916e37edfb8a" path="/var/lib/kubelet/pods/9c3ce5c1-cd13-429e-9734-916e37edfb8a/volumes"
Oct 31 02:15:22.129548 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:22.129527    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfceee3a-9e84-4888-a7ff-bb5fe5b632e5" path="/var/lib/kubelet/pods/dfceee3a-9e84-4888-a7ff-bb5fe5b632e5/volumes"
Oct 31 02:15:22.129975 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:22.129954    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf" path="/var/lib/kubelet/pods/ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf/volumes"
Oct 31 02:15:22.269929 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:22.269898    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7"
Oct 31 02:15:22.270098 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:22.269906    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7" event={"ID":"495060eb-5f49-4f96-bfe7-b89ed0c8b55e","Type":"ContainerDied","Data":"15e0b990b48203a62bd5b9e94cb2296d1011fabb3790edf14370bf660c89b150"}
Oct 31 02:15:22.270098 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:22.270048    5650 scope.go:122] "RemoveContainer" containerID="a088324e52ed68a1ce6d7a21c5171a116e9ddde44473ccd8ec896696f9aaa21c"
Oct 31 02:15:22.331119 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:22.329812    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7"]
Oct 31 02:15:22.334766 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:22.334737    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-6ae519-29890215-vgkc7"]
Oct 31 02:15:22.467049 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:22.467016    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2"
Oct 31 02:15:22.504464 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:22.504418    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql"
Oct 31 02:15:22.574489 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:22.568957    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ae7217-5d7c-4dd3-b830-7e669721ec67-kube-api-access-c8q5d\" (UniqueName: \"kubernetes.io/projected/e0ae7217-5d7c-4dd3-b830-7e669721ec67-kube-api-access-c8q5d\") pod \"e0ae7217-5d7c-4dd3-b830-7e669721ec67\" (UID: \"e0ae7217-5d7c-4dd3-b830-7e669721ec67\") "
Oct 31 02:15:22.574489 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:22.571386    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0ae7217-5d7c-4dd3-b830-7e669721ec67-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0ae7217-5d7c-4dd3-b830-7e669721ec67-maas-api-service-ca\") pod \"e0ae7217-5d7c-4dd3-b830-7e669721ec67\" (UID: \"e0ae7217-5d7c-4dd3-b830-7e669721ec67\") "
Oct 31 02:15:22.574489 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:22.572703    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0ae7217-5d7c-4dd3-b830-7e669721ec67-maas-api-service-ca" pod "e0ae7217-5d7c-4dd3-b830-7e669721ec67" (UID: "e0ae7217-5d7c-4dd3-b830-7e669721ec67"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:22.579409 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:22.575980    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ae7217-5d7c-4dd3-b830-7e669721ec67-kube-api-access-c8q5d" pod "e0ae7217-5d7c-4dd3-b830-7e669721ec67" (UID: "e0ae7217-5d7c-4dd3-b830-7e669721ec67"). InnerVolumeSpecName "kube-api-access-c8q5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:22.673192 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:22.673153    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87ac2c5d-5b80-44ce-904b-713134c4b4f6-kube-api-access-hgs9k\" (UniqueName: \"kubernetes.io/projected/87ac2c5d-5b80-44ce-904b-713134c4b4f6-kube-api-access-hgs9k\") pod \"87ac2c5d-5b80-44ce-904b-713134c4b4f6\" (UID: \"87ac2c5d-5b80-44ce-904b-713134c4b4f6\") "
Oct 31 02:15:22.673355 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:22.673306    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87ac2c5d-5b80-44ce-904b-713134c4b4f6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87ac2c5d-5b80-44ce-904b-713134c4b4f6-maas-api-service-ca\") pod \"87ac2c5d-5b80-44ce-904b-713134c4b4f6\" (UID: \"87ac2c5d-5b80-44ce-904b-713134c4b4f6\") "
Oct 31 02:15:22.673713 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:22.673690    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ac2c5d-5b80-44ce-904b-713134c4b4f6-maas-api-service-ca" pod "87ac2c5d-5b80-44ce-904b-713134c4b4f6" (UID: "87ac2c5d-5b80-44ce-904b-713134c4b4f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:22.673981 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:22.673965    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0ae7217-5d7c-4dd3-b830-7e669721ec67-maas-api-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:22.674031 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:22.673984    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87ac2c5d-5b80-44ce-904b-713134c4b4f6-maas-api-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:22.674031 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:22.673994    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8q5d\" (UniqueName: \"kubernetes.io/projected/e0ae7217-5d7c-4dd3-b830-7e669721ec67-kube-api-access-c8q5d\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:22.675660 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:22.675634    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ac2c5d-5b80-44ce-904b-713134c4b4f6-kube-api-access-hgs9k" pod "87ac2c5d-5b80-44ce-904b-713134c4b4f6" (UID: "87ac2c5d-5b80-44ce-904b-713134c4b4f6"). InnerVolumeSpecName "kube-api-access-hgs9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:22.775607 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:22.775569    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgs9k\" (UniqueName: \"kubernetes.io/projected/87ac2c5d-5b80-44ce-904b-713134c4b4f6-kube-api-access-hgs9k\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:23.286115 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:23.286081    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql"
Oct 31 02:15:23.286115 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:23.286110    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql" event={"ID":"87ac2c5d-5b80-44ce-904b-713134c4b4f6","Type":"ContainerDied","Data":"68c8cab0b14f9bf251ea18023e536588b129d86557fcc6e4f9d10d31115f96da"}
Oct 31 02:15:23.286761 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:23.286173    5650 scope.go:122] "RemoveContainer" containerID="7bd65df32d6904f333653d660f6046b9ac674f25ee8caff55f76c110e31c02d2"
Oct 31 02:15:23.289124 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:23.289086    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2" event={"ID":"e0ae7217-5d7c-4dd3-b830-7e669721ec67","Type":"ContainerDied","Data":"185f47514f57f077fa68b7b5b8d3831cc2ab832c4822ace68374c09db540739f"}
Oct 31 02:15:23.289240 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:23.289098    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2"
Oct 31 02:15:23.299107 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:23.299087    5650 scope.go:122] "RemoveContainer" containerID="95bdd1381f6233480682691ac10f8ecfdbcb0b6123efb245a6a5b599d8ab0611"
Oct 31 02:15:23.314645 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:23.314616    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql"]
Oct 31 02:15:23.322151 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:23.322121    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7a850e-29890215-bffql"]
Oct 31 02:15:23.341077 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:23.341040    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2"]
Oct 31 02:15:23.344941 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:23.344917    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890215-sn5w2"]
Oct 31 02:15:24.121276 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:24.121240    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="495060eb-5f49-4f96-bfe7-b89ed0c8b55e" path="/var/lib/kubelet/pods/495060eb-5f49-4f96-bfe7-b89ed0c8b55e/volumes"
Oct 31 02:15:24.121840 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:24.121822    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87ac2c5d-5b80-44ce-904b-713134c4b4f6" path="/var/lib/kubelet/pods/87ac2c5d-5b80-44ce-904b-713134c4b4f6/volumes"
Oct 31 02:15:24.122260 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:24.122247    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0ae7217-5d7c-4dd3-b830-7e669721ec67" path="/var/lib/kubelet/pods/e0ae7217-5d7c-4dd3-b830-7e669721ec67/volumes"
Oct 31 02:15:28.554124 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:28.554083    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:15:44.360674 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:44.360631    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv"]
Oct 31 02:15:49.362040 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:49.361988    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv" podUID="938e1ac7-e7cd-4773-bce5-305fbf793670" containerName="payload-pre-processing" containerID="cri-o://4d3ff82861e3b26d77c95b919373cbd5629ab6eaadd02ab4839cee050f1005b3" gracePeriod=25
Oct 31 02:15:49.522247 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:49.522214    5650 generic.go:356] "Generic (PLEG): container finished" podID="938e1ac7-e7cd-4773-bce5-305fbf793670" containerID="4d3ff82861e3b26d77c95b919373cbd5629ab6eaadd02ab4839cee050f1005b3" exitCode=0
Oct 31 02:15:49.522497 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:49.522296    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv" event={"ID":"938e1ac7-e7cd-4773-bce5-305fbf793670","Type":"ContainerDied","Data":"4d3ff82861e3b26d77c95b919373cbd5629ab6eaadd02ab4839cee050f1005b3"}
Oct 31 02:15:49.719478 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:49.719425    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv"
Oct 31 02:15:49.799947 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:49.799910    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/938e1ac7-e7cd-4773-bce5-305fbf793670-kube-api-access-jrk87\" (UniqueName: \"kubernetes.io/projected/938e1ac7-e7cd-4773-bce5-305fbf793670-kube-api-access-jrk87\") pod \"938e1ac7-e7cd-4773-bce5-305fbf793670\" (UID: \"938e1ac7-e7cd-4773-bce5-305fbf793670\") "
Oct 31 02:15:49.800148 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:49.800106    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/938e1ac7-e7cd-4773-bce5-305fbf793670-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/938e1ac7-e7cd-4773-bce5-305fbf793670-plugins-config-volume\") pod \"938e1ac7-e7cd-4773-bce5-305fbf793670\" (UID: \"938e1ac7-e7cd-4773-bce5-305fbf793670\") "
Oct 31 02:15:49.800557 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:49.800525    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938e1ac7-e7cd-4773-bce5-305fbf793670-plugins-config-volume" pod "938e1ac7-e7cd-4773-bce5-305fbf793670" (UID: "938e1ac7-e7cd-4773-bce5-305fbf793670"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:15:49.800856 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:49.800835    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/938e1ac7-e7cd-4773-bce5-305fbf793670-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:49.802403 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:49.802376    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938e1ac7-e7cd-4773-bce5-305fbf793670-kube-api-access-jrk87" pod "938e1ac7-e7cd-4773-bce5-305fbf793670" (UID: "938e1ac7-e7cd-4773-bce5-305fbf793670"). InnerVolumeSpecName "kube-api-access-jrk87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:15:49.902282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:49.902179    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrk87\" (UniqueName: \"kubernetes.io/projected/938e1ac7-e7cd-4773-bce5-305fbf793670-kube-api-access-jrk87\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:15:50.535909 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:50.535825    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv"
Oct 31 02:15:50.536352 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:50.535833    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv" event={"ID":"938e1ac7-e7cd-4773-bce5-305fbf793670","Type":"ContainerDied","Data":"27706eb2c1596b41369eb5469a64913c9fa3cfabd27d00d8e664e13f4c1711d1"}
Oct 31 02:15:50.536352 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:50.535975    5650 scope.go:122] "RemoveContainer" containerID="4d3ff82861e3b26d77c95b919373cbd5629ab6eaadd02ab4839cee050f1005b3"
Oct 31 02:15:50.560949 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:50.560915    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv"]
Oct 31 02:15:50.567676 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:50.567646    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e57add-7b6cb574d7-z8fwv"]
Oct 31 02:15:52.121052 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:52.121011    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="938e1ac7-e7cd-4773-bce5-305fbf793670" path="/var/lib/kubelet/pods/938e1ac7-e7cd-4773-bce5-305fbf793670/volumes"
Oct 31 02:15:55.290789 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:55.290749    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"]
Oct 31 02:15:55.291326 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:15:55.290997    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb" podUID="eaaaa634-58c5-4a1f-9eee-816fe2256844" containerName="istio-proxy" containerID="cri-o://f91d7ad3a82a89dffedda938f8a2ef057ff0abbf7423959f3f7748eae1b12a6d" gracePeriod=30
Oct 31 02:15:58.555507 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:15:58.555434    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:16:00.614811 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.614777    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 02:16:00.652158 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.652118    5650 generic.go:356] "Generic (PLEG): container finished" podID="eaaaa634-58c5-4a1f-9eee-816fe2256844" containerID="f91d7ad3a82a89dffedda938f8a2ef057ff0abbf7423959f3f7748eae1b12a6d" exitCode=0
Oct 31 02:16:00.652363 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.652172    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"
Oct 31 02:16:00.652363 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.652197    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb" event={"ID":"eaaaa634-58c5-4a1f-9eee-816fe2256844","Type":"ContainerDied","Data":"f91d7ad3a82a89dffedda938f8a2ef057ff0abbf7423959f3f7748eae1b12a6d"}
Oct 31 02:16:00.652363 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.652262    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb" event={"ID":"eaaaa634-58c5-4a1f-9eee-816fe2256844","Type":"ContainerDied","Data":"fc692c02a1929e6540ea63d48e986ab04eb998239a52f375d77961ffd732bf99"}
Oct 31 02:16:00.652363 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.652287    5650 scope.go:122] "RemoveContainer" containerID="f91d7ad3a82a89dffedda938f8a2ef057ff0abbf7423959f3f7748eae1b12a6d"
Oct 31 02:16:00.653953 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.653925    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-token\" (UniqueName: \"kubernetes.io/projected/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-token\") pod \"eaaaa634-58c5-4a1f-9eee-816fe2256844\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") "
Oct 31 02:16:00.654110 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.654000    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaaaa634-58c5-4a1f-9eee-816fe2256844-kube-api-access-m7bln\" (UniqueName: \"kubernetes.io/projected/eaaaa634-58c5-4a1f-9eee-816fe2256844-kube-api-access-m7bln\") pod \"eaaaa634-58c5-4a1f-9eee-816fe2256844\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") "
Oct 31 02:16:00.654185 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.654126    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-data\") pod \"eaaaa634-58c5-4a1f-9eee-816fe2256844\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") "
Oct 31 02:16:00.654252 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.654237    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaaaa634-58c5-4a1f-9eee-816fe2256844-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaaaa634-58c5-4a1f-9eee-816fe2256844-istiod-ca-cert\") pod \"eaaaa634-58c5-4a1f-9eee-816fe2256844\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") "
Oct 31 02:16:00.654315 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.654277    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-envoy\") pod \"eaaaa634-58c5-4a1f-9eee-816fe2256844\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") "
Oct 31 02:16:00.654315 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.654307    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-workload-certs\") pod \"eaaaa634-58c5-4a1f-9eee-816fe2256844\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") "
Oct 31 02:16:00.654477 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.654424    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-podinfo\") pod \"eaaaa634-58c5-4a1f-9eee-816fe2256844\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") "
Oct 31 02:16:00.654568 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.654481    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-credential-socket\") pod \"eaaaa634-58c5-4a1f-9eee-816fe2256844\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") "
Oct 31 02:16:00.654568 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.654519    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-workload-socket\") pod \"eaaaa634-58c5-4a1f-9eee-816fe2256844\" (UID: \"eaaaa634-58c5-4a1f-9eee-816fe2256844\") "
Oct 31 02:16:00.654681 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.654643    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-data" pod "eaaaa634-58c5-4a1f-9eee-816fe2256844" (UID: "eaaaa634-58c5-4a1f-9eee-816fe2256844"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:00.654892 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.654873    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-workload-certs" pod "eaaaa634-58c5-4a1f-9eee-816fe2256844" (UID: "eaaaa634-58c5-4a1f-9eee-816fe2256844"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:00.655164 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.655144    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaaaa634-58c5-4a1f-9eee-816fe2256844-istiod-ca-cert" pod "eaaaa634-58c5-4a1f-9eee-816fe2256844" (UID: "eaaaa634-58c5-4a1f-9eee-816fe2256844"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:16:00.655513 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.655481    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:00.655619 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.655515    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:00.655619 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.655535    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaaaa634-58c5-4a1f-9eee-816fe2256844-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:00.655750 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.655730    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-credential-socket" pod "eaaaa634-58c5-4a1f-9eee-816fe2256844" (UID: "eaaaa634-58c5-4a1f-9eee-816fe2256844"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:00.655832 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.655762    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-workload-socket" pod "eaaaa634-58c5-4a1f-9eee-816fe2256844" (UID: "eaaaa634-58c5-4a1f-9eee-816fe2256844"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:00.657904 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.657879    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaaaa634-58c5-4a1f-9eee-816fe2256844-kube-api-access-m7bln" pod "eaaaa634-58c5-4a1f-9eee-816fe2256844" (UID: "eaaaa634-58c5-4a1f-9eee-816fe2256844"). InnerVolumeSpecName "kube-api-access-m7bln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:00.658044 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.657901    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-envoy" pod "eaaaa634-58c5-4a1f-9eee-816fe2256844" (UID: "eaaaa634-58c5-4a1f-9eee-816fe2256844"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:00.658044 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.657948    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-token" pod "eaaaa634-58c5-4a1f-9eee-816fe2256844" (UID: "eaaaa634-58c5-4a1f-9eee-816fe2256844"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:00.658240 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.658222    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-podinfo" pod "eaaaa634-58c5-4a1f-9eee-816fe2256844" (UID: "eaaaa634-58c5-4a1f-9eee-816fe2256844"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:16:00.699992 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.699960    5650 scope.go:122] "RemoveContainer" containerID="0806c99cef4632b374470b8fdb2f40f55371bcc91dace685ed0094a0ddfa2857"
Oct 31 02:16:00.712262 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.712231    5650 scope.go:122] "RemoveContainer" containerID="f91d7ad3a82a89dffedda938f8a2ef057ff0abbf7423959f3f7748eae1b12a6d"
Oct 31 02:16:00.712679 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:16:00.712652    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f91d7ad3a82a89dffedda938f8a2ef057ff0abbf7423959f3f7748eae1b12a6d\": container with ID starting with f91d7ad3a82a89dffedda938f8a2ef057ff0abbf7423959f3f7748eae1b12a6d not found: ID does not exist" containerID="f91d7ad3a82a89dffedda938f8a2ef057ff0abbf7423959f3f7748eae1b12a6d"
Oct 31 02:16:00.712773 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.712687    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91d7ad3a82a89dffedda938f8a2ef057ff0abbf7423959f3f7748eae1b12a6d"} err="failed to get container status \"f91d7ad3a82a89dffedda938f8a2ef057ff0abbf7423959f3f7748eae1b12a6d\": rpc error: code = NotFound desc = could not find container \"f91d7ad3a82a89dffedda938f8a2ef057ff0abbf7423959f3f7748eae1b12a6d\": container with ID starting with f91d7ad3a82a89dffedda938f8a2ef057ff0abbf7423959f3f7748eae1b12a6d not found: ID does not exist"
Oct 31 02:16:00.712773 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.712708    5650 scope.go:122] "RemoveContainer" containerID="0806c99cef4632b374470b8fdb2f40f55371bcc91dace685ed0094a0ddfa2857"
Oct 31 02:16:00.713015 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:16:00.712988    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0806c99cef4632b374470b8fdb2f40f55371bcc91dace685ed0094a0ddfa2857\": container with ID starting with 0806c99cef4632b374470b8fdb2f40f55371bcc91dace685ed0094a0ddfa2857 not found: ID does not exist" containerID="0806c99cef4632b374470b8fdb2f40f55371bcc91dace685ed0094a0ddfa2857"
Oct 31 02:16:00.713086 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.713023    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0806c99cef4632b374470b8fdb2f40f55371bcc91dace685ed0094a0ddfa2857"} err="failed to get container status \"0806c99cef4632b374470b8fdb2f40f55371bcc91dace685ed0094a0ddfa2857\": rpc error: code = NotFound desc = could not find container \"0806c99cef4632b374470b8fdb2f40f55371bcc91dace685ed0094a0ddfa2857\": container with ID starting with 0806c99cef4632b374470b8fdb2f40f55371bcc91dace685ed0094a0ddfa2857 not found: ID does not exist"
Oct 31 02:16:00.756818 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.756780    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7bln\" (UniqueName: \"kubernetes.io/projected/eaaaa634-58c5-4a1f-9eee-816fe2256844-kube-api-access-m7bln\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:00.756818 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.756815    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:00.756818 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.756825    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:00.757064 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.756834    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:00.757064 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.756843    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaaaa634-58c5-4a1f-9eee-816fe2256844-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:00.757064 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.756853    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaaaa634-58c5-4a1f-9eee-816fe2256844-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:00.789017 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.788968    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"]
Oct 31 02:16:00.794178 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.794152    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"]
Oct 31 02:16:00.815391 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.815358    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"]
Oct 31 02:16:00.981330 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.981297    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"]
Oct 31 02:16:00.998474 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:00.990003    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e57add-openshift-default-b64556cbf-lmmcb"]
Oct 31 02:16:02.120288 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:02.120251    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaaaa634-58c5-4a1f-9eee-816fe2256844" path="/var/lib/kubelet/pods/eaaaa634-58c5-4a1f-9eee-816fe2256844/volumes"
Oct 31 02:16:15.811870 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:15.811824    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" containerID="cri-o://aef5ec36661ebf6466c8a8db3f616c02506fc750511d60548f92d9d8c6a9fec2" gracePeriod=45
Oct 31 02:16:15.822902 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:15.822866    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main" containerID="cri-o://a0b4763bc801b304d13780d4d24919f48a1a48b1eab6e241afdb1de598fa931f" gracePeriod=45
Oct 31 02:16:15.839265 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:15.838971    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" containerID="cri-o://a7af19eefb39a9079c7e916df5afd8cec2eba54cfcd9a8a18d1e98cdf30494b1" gracePeriod=45
Oct 31 02:16:15.938631 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:15.938595    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": dial tcp 10.134.0.80:8000: connect: connection refused"
Oct 31 02:16:16.155814 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.155783    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 02:16:16.170166 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.170136    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 02:16:16.179805 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.179772    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 02:16:16.351031 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.350939    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c26cdd6-a863-4201-a1e9-f38153be62fc-kube-api-access-mwrfc\" (UniqueName: \"kubernetes.io/projected/0c26cdd6-a863-4201-a1e9-f38153be62fc-kube-api-access-mwrfc\") pod \"0c26cdd6-a863-4201-a1e9-f38153be62fc\" (UID: \"0c26cdd6-a863-4201-a1e9-f38153be62fc\") "
Oct 31 02:16:16.351189 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.351171    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-dshm\") pod \"99608207-5e9f-4565-ae8c-88738a9a6218\" (UID: \"99608207-5e9f-4565-ae8c-88738a9a6218\") "
Oct 31 02:16:16.351248 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.351211    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99608207-5e9f-4565-ae8c-88738a9a6218-tls-certs\" (UniqueName: \"kubernetes.io/secret/99608207-5e9f-4565-ae8c-88738a9a6218-tls-certs\") pod \"99608207-5e9f-4565-ae8c-88738a9a6218\" (UID: \"99608207-5e9f-4565-ae8c-88738a9a6218\") "
Oct 31 02:16:16.351335 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.351316    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-model-cache\") pod \"99608207-5e9f-4565-ae8c-88738a9a6218\" (UID: \"99608207-5e9f-4565-ae8c-88738a9a6218\") "
Oct 31 02:16:16.351585 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.351427    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99608207-5e9f-4565-ae8c-88738a9a6218-kube-api-access-2w67j\" (UniqueName: \"kubernetes.io/projected/99608207-5e9f-4565-ae8c-88738a9a6218-kube-api-access-2w67j\") pod \"99608207-5e9f-4565-ae8c-88738a9a6218\" (UID: \"99608207-5e9f-4565-ae8c-88738a9a6218\") "
Oct 31 02:16:16.351585 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.351504    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-tmp-dir\") pod \"99608207-5e9f-4565-ae8c-88738a9a6218\" (UID: \"99608207-5e9f-4565-ae8c-88738a9a6218\") "
Oct 31 02:16:16.351585 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.351550    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/346efd30-40a6-4b13-b733-7e5d3e19e92f-tls-certs\" (UniqueName: \"kubernetes.io/secret/346efd30-40a6-4b13-b733-7e5d3e19e92f-tls-certs\") pod \"346efd30-40a6-4b13-b733-7e5d3e19e92f\" (UID: \"346efd30-40a6-4b13-b733-7e5d3e19e92f\") "
Oct 31 02:16:16.351856 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.351597    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-model-cache" pod "99608207-5e9f-4565-ae8c-88738a9a6218" (UID: "99608207-5e9f-4565-ae8c-88738a9a6218"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:16.351856 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.351619    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-dshm\") pod \"346efd30-40a6-4b13-b733-7e5d3e19e92f\" (UID: \"346efd30-40a6-4b13-b733-7e5d3e19e92f\") "
Oct 31 02:16:16.351856 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.351663    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/346efd30-40a6-4b13-b733-7e5d3e19e92f-kube-api-access-4xzfn\" (UniqueName: \"kubernetes.io/projected/346efd30-40a6-4b13-b733-7e5d3e19e92f-kube-api-access-4xzfn\") pod \"346efd30-40a6-4b13-b733-7e5d3e19e92f\" (UID: \"346efd30-40a6-4b13-b733-7e5d3e19e92f\") "
Oct 31 02:16:16.351856 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.351699    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-home\" (UniqueName: \"kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-home\") pod \"346efd30-40a6-4b13-b733-7e5d3e19e92f\" (UID: \"346efd30-40a6-4b13-b733-7e5d3e19e92f\") "
Oct 31 02:16:16.351856 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.351727    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-model-cache\") pod \"346efd30-40a6-4b13-b733-7e5d3e19e92f\" (UID: \"346efd30-40a6-4b13-b733-7e5d3e19e92f\") "
Oct 31 02:16:16.351856 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.351761    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-tmp-dir\") pod \"0c26cdd6-a863-4201-a1e9-f38153be62fc\" (UID: \"0c26cdd6-a863-4201-a1e9-f38153be62fc\") "
Oct 31 02:16:16.351856 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.351789    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-model-cache\") pod \"0c26cdd6-a863-4201-a1e9-f38153be62fc\" (UID: \"0c26cdd6-a863-4201-a1e9-f38153be62fc\") "
Oct 31 02:16:16.352233 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.351876    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c26cdd6-a863-4201-a1e9-f38153be62fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c26cdd6-a863-4201-a1e9-f38153be62fc-tls-certs\") pod \"0c26cdd6-a863-4201-a1e9-f38153be62fc\" (UID: \"0c26cdd6-a863-4201-a1e9-f38153be62fc\") "
Oct 31 02:16:16.352233 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.351910    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-home\" (UniqueName: \"kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-home\") pod \"0c26cdd6-a863-4201-a1e9-f38153be62fc\" (UID: \"0c26cdd6-a863-4201-a1e9-f38153be62fc\") "
Oct 31 02:16:16.352233 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.351940    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-dshm\") pod \"0c26cdd6-a863-4201-a1e9-f38153be62fc\" (UID: \"0c26cdd6-a863-4201-a1e9-f38153be62fc\") "
Oct 31 02:16:16.352233 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.352014    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-home\" (UniqueName: \"kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-home\") pod \"99608207-5e9f-4565-ae8c-88738a9a6218\" (UID: \"99608207-5e9f-4565-ae8c-88738a9a6218\") "
Oct 31 02:16:16.352233 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.352052    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-tmp-dir\") pod \"346efd30-40a6-4b13-b733-7e5d3e19e92f\" (UID: \"346efd30-40a6-4b13-b733-7e5d3e19e92f\") "
Oct 31 02:16:16.352845 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.352319    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-tmp-dir" pod "99608207-5e9f-4565-ae8c-88738a9a6218" (UID: "99608207-5e9f-4565-ae8c-88738a9a6218"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:16.352908 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.352892    5650 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-model-cache\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:16.352970 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.352917    5650 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-tmp-dir\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:16.353386 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.353330    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-home" pod "346efd30-40a6-4b13-b733-7e5d3e19e92f" (UID: "346efd30-40a6-4b13-b733-7e5d3e19e92f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:16.354665 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.353401    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-tmp-dir" pod "346efd30-40a6-4b13-b733-7e5d3e19e92f" (UID: "346efd30-40a6-4b13-b733-7e5d3e19e92f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:16.354665 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.353674    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-home" pod "99608207-5e9f-4565-ae8c-88738a9a6218" (UID: "99608207-5e9f-4565-ae8c-88738a9a6218"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:16.354665 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.353682    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-model-cache" pod "0c26cdd6-a863-4201-a1e9-f38153be62fc" (UID: "0c26cdd6-a863-4201-a1e9-f38153be62fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:16.354665 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.353965    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-tmp-dir" pod "0c26cdd6-a863-4201-a1e9-f38153be62fc" (UID: "0c26cdd6-a863-4201-a1e9-f38153be62fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:16.354665 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.354432    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-home" pod "0c26cdd6-a863-4201-a1e9-f38153be62fc" (UID: "0c26cdd6-a863-4201-a1e9-f38153be62fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:16.357399 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.354946    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99608207-5e9f-4565-ae8c-88738a9a6218-tls-certs" pod "99608207-5e9f-4565-ae8c-88738a9a6218" (UID: "99608207-5e9f-4565-ae8c-88738a9a6218"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:16:16.357399 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.356846    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c26cdd6-a863-4201-a1e9-f38153be62fc-kube-api-access-mwrfc" pod "0c26cdd6-a863-4201-a1e9-f38153be62fc" (UID: "0c26cdd6-a863-4201-a1e9-f38153be62fc"). InnerVolumeSpecName "kube-api-access-mwrfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:16.357399 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.356932    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-model-cache" pod "346efd30-40a6-4b13-b733-7e5d3e19e92f" (UID: "346efd30-40a6-4b13-b733-7e5d3e19e92f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:16.357399 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.356984    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-dshm" pod "99608207-5e9f-4565-ae8c-88738a9a6218" (UID: "99608207-5e9f-4565-ae8c-88738a9a6218"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:16.357978 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.357940    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/346efd30-40a6-4b13-b733-7e5d3e19e92f-tls-certs" pod "346efd30-40a6-4b13-b733-7e5d3e19e92f" (UID: "346efd30-40a6-4b13-b733-7e5d3e19e92f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:16:16.359179 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.358157    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99608207-5e9f-4565-ae8c-88738a9a6218-kube-api-access-2w67j" pod "99608207-5e9f-4565-ae8c-88738a9a6218" (UID: "99608207-5e9f-4565-ae8c-88738a9a6218"). InnerVolumeSpecName "kube-api-access-2w67j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:16.359179 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.358497    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-dshm" pod "0c26cdd6-a863-4201-a1e9-f38153be62fc" (UID: "0c26cdd6-a863-4201-a1e9-f38153be62fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:16.359179 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.358745    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c26cdd6-a863-4201-a1e9-f38153be62fc-tls-certs" pod "0c26cdd6-a863-4201-a1e9-f38153be62fc" (UID: "0c26cdd6-a863-4201-a1e9-f38153be62fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:16:16.361803 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.359762    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/346efd30-40a6-4b13-b733-7e5d3e19e92f-kube-api-access-4xzfn" pod "346efd30-40a6-4b13-b733-7e5d3e19e92f" (UID: "346efd30-40a6-4b13-b733-7e5d3e19e92f"). InnerVolumeSpecName "kube-api-access-4xzfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:16.365768 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.365627    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-dshm" pod "346efd30-40a6-4b13-b733-7e5d3e19e92f" (UID: "346efd30-40a6-4b13-b733-7e5d3e19e92f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:16.454073 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.454035    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwrfc\" (UniqueName: \"kubernetes.io/projected/0c26cdd6-a863-4201-a1e9-f38153be62fc-kube-api-access-mwrfc\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:16.454073 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.454071    5650 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-dshm\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:16.454073 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.454081    5650 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99608207-5e9f-4565-ae8c-88738a9a6218-tls-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:16.454329 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.454090    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w67j\" (UniqueName: \"kubernetes.io/projected/99608207-5e9f-4565-ae8c-88738a9a6218-kube-api-access-2w67j\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:16.454329 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.454099    5650 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/346efd30-40a6-4b13-b733-7e5d3e19e92f-tls-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:16.454329 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.454108    5650 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-dshm\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:16.454329 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.454115    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xzfn\" (UniqueName: \"kubernetes.io/projected/346efd30-40a6-4b13-b733-7e5d3e19e92f-kube-api-access-4xzfn\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:16.454329 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.454123    5650 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-home\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:16.454329 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.454133    5650 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-model-cache\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:16.454329 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.454141    5650 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-tmp-dir\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:16.454329 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.454149    5650 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-model-cache\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:16.454329 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.454157    5650 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c26cdd6-a863-4201-a1e9-f38153be62fc-tls-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:16.454329 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.454165    5650 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-home\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:16.454329 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.454172    5650 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c26cdd6-a863-4201-a1e9-f38153be62fc-dshm\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:16.454329 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.454181    5650 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99608207-5e9f-4565-ae8c-88738a9a6218-home\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:16.454329 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.454189    5650 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/346efd30-40a6-4b13-b733-7e5d3e19e92f-tmp-dir\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:16.825714 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.825676    5650 generic.go:356] "Generic (PLEG): container finished" podID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerID="aef5ec36661ebf6466c8a8db3f616c02506fc750511d60548f92d9d8c6a9fec2" exitCode=0
Oct 31 02:16:16.826302 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.825742    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"
Oct 31 02:16:16.826302 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.825763    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" event={"ID":"0c26cdd6-a863-4201-a1e9-f38153be62fc","Type":"ContainerDied","Data":"aef5ec36661ebf6466c8a8db3f616c02506fc750511d60548f92d9d8c6a9fec2"}
Oct 31 02:16:16.826302 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.825816    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn" event={"ID":"0c26cdd6-a863-4201-a1e9-f38153be62fc","Type":"ContainerDied","Data":"ac2c7c9f9ae61fb2d73ba5700c1ad4003680c389121465f0eced134a6d908165"}
Oct 31 02:16:16.826302 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.825840    5650 scope.go:122] "RemoveContainer" containerID="aef5ec36661ebf6466c8a8db3f616c02506fc750511d60548f92d9d8c6a9fec2"
Oct 31 02:16:16.827780 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.827751    5650 generic.go:356] "Generic (PLEG): container finished" podID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerID="a0b4763bc801b304d13780d4d24919f48a1a48b1eab6e241afdb1de598fa931f" exitCode=0
Oct 31 02:16:16.827904 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.827801    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"
Oct 31 02:16:16.827904 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.827877    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" event={"ID":"346efd30-40a6-4b13-b733-7e5d3e19e92f","Type":"ContainerDied","Data":"a0b4763bc801b304d13780d4d24919f48a1a48b1eab6e241afdb1de598fa931f"}
Oct 31 02:16:16.828019 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.827923    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w" event={"ID":"346efd30-40a6-4b13-b733-7e5d3e19e92f","Type":"ContainerDied","Data":"d6d28560c0dfc033b64926982ed2ba2bba48d5a89fa6ff700504df54f69350d4"}
Oct 31 02:16:16.832107 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.832077    5650 generic.go:356] "Generic (PLEG): container finished" podID="99608207-5e9f-4565-ae8c-88738a9a6218" containerID="a7af19eefb39a9079c7e916df5afd8cec2eba54cfcd9a8a18d1e98cdf30494b1" exitCode=0
Oct 31 02:16:16.832230 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.832127    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" event={"ID":"99608207-5e9f-4565-ae8c-88738a9a6218","Type":"ContainerDied","Data":"a7af19eefb39a9079c7e916df5afd8cec2eba54cfcd9a8a18d1e98cdf30494b1"}
Oct 31 02:16:16.832230 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.832158    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" event={"ID":"99608207-5e9f-4565-ae8c-88738a9a6218","Type":"ContainerDied","Data":"f9f7a08389a418ab94cf6724ac5530f834c833d0ffa7f8b511a9d04bac16836e"}
Oct 31 02:16:16.832230 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.832221    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"
Oct 31 02:16:16.865049 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.865022    5650 scope.go:122] "RemoveContainer" containerID="aef5ec36661ebf6466c8a8db3f616c02506fc750511d60548f92d9d8c6a9fec2"
Oct 31 02:16:16.865389 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:16:16.865366    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aef5ec36661ebf6466c8a8db3f616c02506fc750511d60548f92d9d8c6a9fec2\": container with ID starting with aef5ec36661ebf6466c8a8db3f616c02506fc750511d60548f92d9d8c6a9fec2 not found: ID does not exist" containerID="aef5ec36661ebf6466c8a8db3f616c02506fc750511d60548f92d9d8c6a9fec2"
Oct 31 02:16:16.865507 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.865407    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aef5ec36661ebf6466c8a8db3f616c02506fc750511d60548f92d9d8c6a9fec2"} err="failed to get container status \"aef5ec36661ebf6466c8a8db3f616c02506fc750511d60548f92d9d8c6a9fec2\": rpc error: code = NotFound desc = could not find container \"aef5ec36661ebf6466c8a8db3f616c02506fc750511d60548f92d9d8c6a9fec2\": container with ID starting with aef5ec36661ebf6466c8a8db3f616c02506fc750511d60548f92d9d8c6a9fec2 not found: ID does not exist"
Oct 31 02:16:16.865507 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.865437    5650 scope.go:122] "RemoveContainer" containerID="a0b4763bc801b304d13780d4d24919f48a1a48b1eab6e241afdb1de598fa931f"
Oct 31 02:16:16.877382 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.877357    5650 scope.go:122] "RemoveContainer" containerID="a0b4763bc801b304d13780d4d24919f48a1a48b1eab6e241afdb1de598fa931f"
Oct 31 02:16:16.878600 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:16:16.878572    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0b4763bc801b304d13780d4d24919f48a1a48b1eab6e241afdb1de598fa931f\": container with ID starting with a0b4763bc801b304d13780d4d24919f48a1a48b1eab6e241afdb1de598fa931f not found: ID does not exist" containerID="a0b4763bc801b304d13780d4d24919f48a1a48b1eab6e241afdb1de598fa931f"
Oct 31 02:16:16.878701 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.878612    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0b4763bc801b304d13780d4d24919f48a1a48b1eab6e241afdb1de598fa931f"} err="failed to get container status \"a0b4763bc801b304d13780d4d24919f48a1a48b1eab6e241afdb1de598fa931f\": rpc error: code = NotFound desc = could not find container \"a0b4763bc801b304d13780d4d24919f48a1a48b1eab6e241afdb1de598fa931f\": container with ID starting with a0b4763bc801b304d13780d4d24919f48a1a48b1eab6e241afdb1de598fa931f not found: ID does not exist"
Oct 31 02:16:16.878701 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.878638    5650 scope.go:122] "RemoveContainer" containerID="a7af19eefb39a9079c7e916df5afd8cec2eba54cfcd9a8a18d1e98cdf30494b1"
Oct 31 02:16:16.882809 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.882779    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"]
Oct 31 02:16:16.887713 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.887682    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/distinct-2-w0-3cf6f4-kserve-7fb8b64c78-9xpbn"]
Oct 31 02:16:16.892234 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.892212    5650 scope.go:122] "RemoveContainer" containerID="a7af19eefb39a9079c7e916df5afd8cec2eba54cfcd9a8a18d1e98cdf30494b1"
Oct 31 02:16:16.892654 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:16:16.892635    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7af19eefb39a9079c7e916df5afd8cec2eba54cfcd9a8a18d1e98cdf30494b1\": container with ID starting with a7af19eefb39a9079c7e916df5afd8cec2eba54cfcd9a8a18d1e98cdf30494b1 not found: ID does not exist" containerID="a7af19eefb39a9079c7e916df5afd8cec2eba54cfcd9a8a18d1e98cdf30494b1"
Oct 31 02:16:16.892708 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.892663    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7af19eefb39a9079c7e916df5afd8cec2eba54cfcd9a8a18d1e98cdf30494b1"} err="failed to get container status \"a7af19eefb39a9079c7e916df5afd8cec2eba54cfcd9a8a18d1e98cdf30494b1\": rpc error: code = NotFound desc = could not find container \"a7af19eefb39a9079c7e916df5afd8cec2eba54cfcd9a8a18d1e98cdf30494b1\": container with ID starting with a7af19eefb39a9079c7e916df5afd8cec2eba54cfcd9a8a18d1e98cdf30494b1 not found: ID does not exist"
Oct 31 02:16:16.899889 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.899857    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"]
Oct 31 02:16:16.905459 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:16:16.905413    5650 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod346efd30_40a6_4b13_b733_7e5d3e19e92f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99608207_5e9f_4565_ae8c_88738a9a6218.slice/crio-f9f7a08389a418ab94cf6724ac5530f834c833d0ffa7f8b511a9d04bac16836e\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99608207_5e9f_4565_ae8c_88738a9a6218.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 02:16:16.905801 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.905778    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/distinct-w0-3cf6f4-kserve-64b454cb8f-rv84w"]
Oct 31 02:16:16.924246 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.924202    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"]
Oct 31 02:16:16.928564 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:16.928532    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9"]
Oct 31 02:16:17.161908 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:17.161819    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/simulator-w0-3cf6f4-kserve-8479dd7567-zknb9" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:16:18.121119 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:18.121083    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" path="/var/lib/kubelet/pods/0c26cdd6-a863-4201-a1e9-f38153be62fc/volumes"
Oct 31 02:16:18.121661 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:18.121647    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" path="/var/lib/kubelet/pods/346efd30-40a6-4b13-b733-7e5d3e19e92f/volumes"
Oct 31 02:16:18.122082 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:18.122068    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" path="/var/lib/kubelet/pods/99608207-5e9f-4565-ae8c-88738a9a6218/volumes"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.718410    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e50821-qlwlt"]
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.719849    5650 cpu_manager.go:414] "containerMap: removing container" podUID="f6566d82-f8bc-44f8-b4ff-21edc8e67810" containerName="cleanup"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.719874    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6566d82-f8bc-44f8-b4ff-21edc8e67810" containerName="cleanup"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.719895    5650 cpu_manager.go:414] "containerMap: removing container" podUID="f6566d82-f8bc-44f8-b4ff-21edc8e67810" containerName="cleanup"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.719903    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6566d82-f8bc-44f8-b4ff-21edc8e67810" containerName="cleanup"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.719919    5650 cpu_manager.go:414] "containerMap: removing container" podUID="e0ae7217-5d7c-4dd3-b830-7e669721ec67" containerName="cleanup"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.719929    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ae7217-5d7c-4dd3-b830-7e669721ec67" containerName="cleanup"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.719945    5650 cpu_manager.go:414] "containerMap: removing container" podUID="ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf" containerName="cleanup"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.719954    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf" containerName="cleanup"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.719967    5650 cpu_manager.go:414] "containerMap: removing container" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.719974    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.719983    5650 cpu_manager.go:414] "containerMap: removing container" podUID="6f0840a2-aff1-4bb6-bba8-968c0ee0e277" containerName="payload-pre-processing"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.719990    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0840a2-aff1-4bb6-bba8-968c0ee0e277" containerName="payload-pre-processing"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720004    5650 cpu_manager.go:414] "containerMap: removing container" podUID="6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d" containerName="cleanup"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720010    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d" containerName="cleanup"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720023    5650 cpu_manager.go:414] "containerMap: removing container" podUID="6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d" containerName="cleanup"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720033    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d" containerName="cleanup"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720049    5650 cpu_manager.go:414] "containerMap: removing container" podUID="dfceee3a-9e84-4888-a7ff-bb5fe5b632e5" containerName="cleanup"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720057    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfceee3a-9e84-4888-a7ff-bb5fe5b632e5" containerName="cleanup"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720067    5650 cpu_manager.go:414] "containerMap: removing container" podUID="e0fa4248-e818-450c-bf1b-74f859787788" containerName="istio-proxy"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720076    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0fa4248-e818-450c-bf1b-74f859787788" containerName="istio-proxy"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720097    5650 cpu_manager.go:414] "containerMap: removing container" podUID="9c3ce5c1-cd13-429e-9734-916e37edfb8a" containerName="cleanup"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720105    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c3ce5c1-cd13-429e-9734-916e37edfb8a" containerName="cleanup"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720115    5650 cpu_manager.go:414] "containerMap: removing container" podUID="0c8cd351-2df1-4f99-a782-e63b9aab297f" containerName="cleanup"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720124    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c8cd351-2df1-4f99-a782-e63b9aab297f" containerName="cleanup"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720142    5650 cpu_manager.go:414] "containerMap: removing container" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720149    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720161    5650 cpu_manager.go:414] "containerMap: removing container" podUID="87ac2c5d-5b80-44ce-904b-713134c4b4f6" containerName="cleanup"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720168    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ac2c5d-5b80-44ce-904b-713134c4b4f6" containerName="cleanup"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720179    5650 cpu_manager.go:414] "containerMap: removing container" podUID="55d0de06-2458-43fb-bf56-f02492a67dea" containerName="payload-pre-processing"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720186    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55d0de06-2458-43fb-bf56-f02492a67dea" containerName="payload-pre-processing"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720215    5650 cpu_manager.go:414] "containerMap: removing container" podUID="495060eb-5f49-4f96-bfe7-b89ed0c8b55e" containerName="cleanup"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720223    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="495060eb-5f49-4f96-bfe7-b89ed0c8b55e" containerName="cleanup"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720243    5650 cpu_manager.go:414] "containerMap: removing container" podUID="eaaaa634-58c5-4a1f-9eee-816fe2256844" containerName="istio-proxy"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720251    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaaaa634-58c5-4a1f-9eee-816fe2256844" containerName="istio-proxy"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720263    5650 cpu_manager.go:414] "containerMap: removing container" podUID="495060eb-5f49-4f96-bfe7-b89ed0c8b55e" containerName="cleanup"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720269    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="495060eb-5f49-4f96-bfe7-b89ed0c8b55e" containerName="cleanup"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720283    5650 cpu_manager.go:414] "containerMap: removing container" podUID="eaaaa634-58c5-4a1f-9eee-816fe2256844" containerName="istio-proxy"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720291    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaaaa634-58c5-4a1f-9eee-816fe2256844" containerName="istio-proxy"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720303    5650 cpu_manager.go:414] "containerMap: removing container" podUID="87ac2c5d-5b80-44ce-904b-713134c4b4f6" containerName="cleanup"
Oct 31 02:16:19.720282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720311    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ac2c5d-5b80-44ce-904b-713134c4b4f6" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720327    5650 cpu_manager.go:414] "containerMap: removing container" podUID="9c3ce5c1-cd13-429e-9734-916e37edfb8a" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720333    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c3ce5c1-cd13-429e-9734-916e37edfb8a" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720344    5650 cpu_manager.go:414] "containerMap: removing container" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720351    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720369    5650 cpu_manager.go:414] "containerMap: removing container" podUID="0c8cd351-2df1-4f99-a782-e63b9aab297f" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720376    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c8cd351-2df1-4f99-a782-e63b9aab297f" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720386    5650 cpu_manager.go:414] "containerMap: removing container" podUID="dfceee3a-9e84-4888-a7ff-bb5fe5b632e5" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720393    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfceee3a-9e84-4888-a7ff-bb5fe5b632e5" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720407    5650 cpu_manager.go:414] "containerMap: removing container" podUID="0c8cd351-2df1-4f99-a782-e63b9aab297f" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720413    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c8cd351-2df1-4f99-a782-e63b9aab297f" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720430    5650 cpu_manager.go:414] "containerMap: removing container" podUID="495060eb-5f49-4f96-bfe7-b89ed0c8b55e" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720437    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="495060eb-5f49-4f96-bfe7-b89ed0c8b55e" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720465    5650 cpu_manager.go:414] "containerMap: removing container" podUID="e0ae7217-5d7c-4dd3-b830-7e669721ec67" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720472    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ae7217-5d7c-4dd3-b830-7e669721ec67" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720489    5650 cpu_manager.go:414] "containerMap: removing container" podUID="938e1ac7-e7cd-4773-bce5-305fbf793670" containerName="payload-pre-processing"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720495    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="938e1ac7-e7cd-4773-bce5-305fbf793670" containerName="payload-pre-processing"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720556    5650 cpu_manager.go:414] "containerMap: removing container" podUID="ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720568    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720753    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720769    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6566d82-f8bc-44f8-b4ff-21edc8e67810" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720783    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6566d82-f8bc-44f8-b4ff-21edc8e67810" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720794    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6566d82-f8bc-44f8-b4ff-21edc8e67810" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720805    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ae7217-5d7c-4dd3-b830-7e669721ec67" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720814    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720825    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="346efd30-40a6-4b13-b733-7e5d3e19e92f" containerName="main"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720835    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0840a2-aff1-4bb6-bba8-968c0ee0e277" containerName="payload-pre-processing"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720843    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720853    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720864    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfceee3a-9e84-4888-a7ff-bb5fe5b632e5" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720871    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0fa4248-e818-450c-bf1b-74f859787788" containerName="istio-proxy"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720880    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c3ce5c1-cd13-429e-9734-916e37edfb8a" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720889    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c8cd351-2df1-4f99-a782-e63b9aab297f" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720900    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c26cdd6-a863-4201-a1e9-f38153be62fc" containerName="main"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720915    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ac2c5d-5b80-44ce-904b-713134c4b4f6" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720924    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="55d0de06-2458-43fb-bf56-f02492a67dea" containerName="payload-pre-processing"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720935    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="495060eb-5f49-4f96-bfe7-b89ed0c8b55e" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720943    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfceee3a-9e84-4888-a7ff-bb5fe5b632e5" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720953    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaaaa634-58c5-4a1f-9eee-816fe2256844" containerName="istio-proxy"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720962    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ac2c5d-5b80-44ce-904b-713134c4b4f6" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720970    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaaaa634-58c5-4a1f-9eee-816fe2256844" containerName="istio-proxy"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720977    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ac2c5d-5b80-44ce-904b-713134c4b4f6" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720987    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c3ce5c1-cd13-429e-9734-916e37edfb8a" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.720994    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="99608207-5e9f-4565-ae8c-88738a9a6218" containerName="main"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.721006    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfceee3a-9e84-4888-a7ff-bb5fe5b632e5" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.721014    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c8cd351-2df1-4f99-a782-e63b9aab297f" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.721023    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="495060eb-5f49-4f96-bfe7-b89ed0c8b55e" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.721033    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ae7217-5d7c-4dd3-b830-7e669721ec67" containerName="cleanup"
Oct 31 02:16:19.722928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.721043    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="938e1ac7-e7cd-4773-bce5-305fbf793670" containerName="payload-pre-processing"
Oct 31 02:16:19.726992 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.726969    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e50821-qlwlt"
Oct 31 02:16:19.733617 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.733336    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:16:19.733617 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.733555    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:16:19.736871 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.736822    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e50821-qlwlt"]
Oct 31 02:16:19.831941 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.831905    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96e50821-qlwlt\" (UID: \"283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e50821-qlwlt"
Oct 31 02:16:19.832109 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.832078    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c8rj\" (UniqueName: \"kubernetes.io/projected/283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5-kube-api-access-8c8rj\") pod \"maas-api-revoke-sub-96e50821-qlwlt\" (UID: \"283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e50821-qlwlt"
Oct 31 02:16:19.832229 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.832208    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96e50821-qlwlt\" (UID: \"283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e50821-qlwlt"
Oct 31 02:16:19.933350 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.933293    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c8rj\" (UniqueName: \"kubernetes.io/projected/283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5-kube-api-access-8c8rj\") pod \"maas-api-revoke-sub-96e50821-qlwlt\" (UID: \"283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e50821-qlwlt"
Oct 31 02:16:19.933583 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.933483    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96e50821-qlwlt\" (UID: \"283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e50821-qlwlt"
Oct 31 02:16:19.933687 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.933668    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96e50821-qlwlt\" (UID: \"283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e50821-qlwlt"
Oct 31 02:16:19.934278 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.934260    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96e50821-qlwlt\" (UID: \"283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e50821-qlwlt"
Oct 31 02:16:19.935886 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.935856    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96e50821-qlwlt\" (UID: \"283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e50821-qlwlt"
Oct 31 02:16:19.946394 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:19.946360    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c8rj\" (UniqueName: \"kubernetes.io/projected/283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5-kube-api-access-8c8rj\") pod \"maas-api-revoke-sub-96e50821-qlwlt\" (UID: \"283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e50821-qlwlt"
Oct 31 02:16:20.057337 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:20.057227    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e50821-qlwlt"
Oct 31 02:16:20.388771 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:20.388738    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e50821-qlwlt"]
Oct 31 02:16:20.389871 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:16:20.389836    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod283f8e1a_8fae_4a8a_b75a_8fbdd5e563b5.slice/crio-2d4592f9eeb5b47e5fcc7e5427123ec3592b715b230b86e4230844e13bb56c33 WatchSource:0}: Error finding container 2d4592f9eeb5b47e5fcc7e5427123ec3592b715b230b86e4230844e13bb56c33: Status 404 returned error can't find the container with id 2d4592f9eeb5b47e5fcc7e5427123ec3592b715b230b86e4230844e13bb56c33
Oct 31 02:16:20.947550 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:20.947329    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e50821-qlwlt" event={"ID":"283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5","Type":"ContainerStarted","Data":"b7dd9f62d3010cdb05741805b8f80a0484755945161f6ee8d818d17fb9a290fa"}
Oct 31 02:16:20.947550 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:20.947398    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e50821-qlwlt" event={"ID":"283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5","Type":"ContainerStarted","Data":"2d4592f9eeb5b47e5fcc7e5427123ec3592b715b230b86e4230844e13bb56c33"}
Oct 31 02:16:22.003482 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:22.000917    5650 generic.go:356] "Generic (PLEG): container finished" podID="283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5" containerID="b7dd9f62d3010cdb05741805b8f80a0484755945161f6ee8d818d17fb9a290fa" exitCode=0
Oct 31 02:16:22.003482 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:22.000973    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e50821-qlwlt" event={"ID":"283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5","Type":"ContainerDied","Data":"b7dd9f62d3010cdb05741805b8f80a0484755945161f6ee8d818d17fb9a290fa"}
Oct 31 02:16:23.221278 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:23.221220    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e50821-qlwlt"
Oct 31 02:16:23.317282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:23.317195    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5-maas-api-xks-ca\") pod \"283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5\" (UID: \"283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5\") "
Oct 31 02:16:23.317487 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:23.317289    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5-maas-api-openshift-ca\") pod \"283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5\" (UID: \"283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5\") "
Oct 31 02:16:23.317487 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:23.317424    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5-kube-api-access-8c8rj\" (UniqueName: \"kubernetes.io/projected/283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5-kube-api-access-8c8rj\") pod \"283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5\" (UID: \"283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5\") "
Oct 31 02:16:23.317739 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:23.317709    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5-maas-api-openshift-ca" pod "283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5" (UID: "283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:16:23.318100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:23.318075    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:23.319778 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:23.319753    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5-maas-api-xks-ca" pod "283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5" (UID: "283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:16:23.319938 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:23.319914    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5-kube-api-access-8c8rj" pod "283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5" (UID: "283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5"). InnerVolumeSpecName "kube-api-access-8c8rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:23.419545 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:23.419508    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:23.419545 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:23.419541    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c8rj\" (UniqueName: \"kubernetes.io/projected/283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5-kube-api-access-8c8rj\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:24.019923 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:24.019889    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e50821-qlwlt"
Oct 31 02:16:24.020127 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:24.019919    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e50821-qlwlt" event={"ID":"283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5","Type":"ContainerDied","Data":"2d4592f9eeb5b47e5fcc7e5427123ec3592b715b230b86e4230844e13bb56c33"}
Oct 31 02:16:24.020127 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:24.019979    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d4592f9eeb5b47e5fcc7e5427123ec3592b715b230b86e4230844e13bb56c33"
Oct 31 02:16:28.559322 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:16:28.559236    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:16:42.685748 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:42.685665    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"]
Oct 31 02:16:42.713481 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:42.710675    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv" podUID="7f8ad70f-aced-426e-8d7a-d1c17c256582" containerName="istio-proxy" containerID="cri-o://295c7f3229683f3fc00cfd7de2ef7d1dad2e479f9e9bbdbfa89688581af56a44" gracePeriod=30
Oct 31 02:16:43.651312 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:43.651275    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk"]
Oct 31 02:16:43.674551 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:43.674515    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl"]
Oct 31 02:16:45.476479 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:45.476406    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw"]
Oct 31 02:16:45.477050 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:45.476700    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw" podUID="dd57354a-e4f0-42fc-a537-336d346ffee0" containerName="maas-api" containerID="cri-o://0d309825e2abc9cd62b6a566bef818ade7733278d160f0f9fe7b79695dfa77d9" gracePeriod=30
Oct 31 02:16:45.981282 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:45.981248    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw"
Oct 31 02:16:46.126325 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:46.126199    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd57354a-e4f0-42fc-a537-336d346ffee0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd57354a-e4f0-42fc-a537-336d346ffee0-openshift-service-ca\") pod \"dd57354a-e4f0-42fc-a537-336d346ffee0\" (UID: \"dd57354a-e4f0-42fc-a537-336d346ffee0\") "
Oct 31 02:16:46.126325 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:46.126275    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd57354a-e4f0-42fc-a537-336d346ffee0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd57354a-e4f0-42fc-a537-336d346ffee0-metrics-certs\") pod \"dd57354a-e4f0-42fc-a537-336d346ffee0\" (UID: \"dd57354a-e4f0-42fc-a537-336d346ffee0\") "
Oct 31 02:16:46.126751 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:46.126495    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd57354a-e4f0-42fc-a537-336d346ffee0-kube-api-access-nczwv\" (UniqueName: \"kubernetes.io/projected/dd57354a-e4f0-42fc-a537-336d346ffee0-kube-api-access-nczwv\") pod \"dd57354a-e4f0-42fc-a537-336d346ffee0\" (UID: \"dd57354a-e4f0-42fc-a537-336d346ffee0\") "
Oct 31 02:16:46.126751 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:46.126537    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd57354a-e4f0-42fc-a537-336d346ffee0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd57354a-e4f0-42fc-a537-336d346ffee0-maas-api-tls\") pod \"dd57354a-e4f0-42fc-a537-336d346ffee0\" (UID: \"dd57354a-e4f0-42fc-a537-336d346ffee0\") "
Oct 31 02:16:46.126751 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:46.126633    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd57354a-e4f0-42fc-a537-336d346ffee0-openshift-service-ca" pod "dd57354a-e4f0-42fc-a537-336d346ffee0" (UID: "dd57354a-e4f0-42fc-a537-336d346ffee0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:16:46.127340 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:46.127316    5650 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd57354a-e4f0-42fc-a537-336d346ffee0-openshift-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:46.129854 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:46.129826    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd57354a-e4f0-42fc-a537-336d346ffee0-metrics-certs" pod "dd57354a-e4f0-42fc-a537-336d346ffee0" (UID: "dd57354a-e4f0-42fc-a537-336d346ffee0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:16:46.129954 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:46.129850    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd57354a-e4f0-42fc-a537-336d346ffee0-maas-api-tls" pod "dd57354a-e4f0-42fc-a537-336d346ffee0" (UID: "dd57354a-e4f0-42fc-a537-336d346ffee0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:16:46.129954 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:46.129859    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd57354a-e4f0-42fc-a537-336d346ffee0-kube-api-access-nczwv" pod "dd57354a-e4f0-42fc-a537-336d346ffee0" (UID: "dd57354a-e4f0-42fc-a537-336d346ffee0"). InnerVolumeSpecName "kube-api-access-nczwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:46.235815 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:46.229482    5650 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd57354a-e4f0-42fc-a537-336d346ffee0-metrics-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:46.235815 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:46.229547    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nczwv\" (UniqueName: \"kubernetes.io/projected/dd57354a-e4f0-42fc-a537-336d346ffee0-kube-api-access-nczwv\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:46.235815 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:46.229572    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd57354a-e4f0-42fc-a537-336d346ffee0-maas-api-tls\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:46.494963 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:46.494927    5650 generic.go:356] "Generic (PLEG): container finished" podID="dd57354a-e4f0-42fc-a537-336d346ffee0" containerID="0d309825e2abc9cd62b6a566bef818ade7733278d160f0f9fe7b79695dfa77d9" exitCode=0
Oct 31 02:16:46.495429 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:46.494969    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw" event={"ID":"dd57354a-e4f0-42fc-a537-336d346ffee0","Type":"ContainerDied","Data":"0d309825e2abc9cd62b6a566bef818ade7733278d160f0f9fe7b79695dfa77d9"}
Oct 31 02:16:46.495429 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:46.494999    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw"
Oct 31 02:16:46.495429 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:46.495038    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw" event={"ID":"dd57354a-e4f0-42fc-a537-336d346ffee0","Type":"ContainerDied","Data":"ea899212c01da9ab04f4a84fd02ed66659764874822a038d848209a3dd5499d2"}
Oct 31 02:16:46.495429 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:46.495059    5650 scope.go:122] "RemoveContainer" containerID="0d309825e2abc9cd62b6a566bef818ade7733278d160f0f9fe7b79695dfa77d9"
Oct 31 02:16:46.512336 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:46.512314    5650 scope.go:122] "RemoveContainer" containerID="0d309825e2abc9cd62b6a566bef818ade7733278d160f0f9fe7b79695dfa77d9"
Oct 31 02:16:46.512705 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:16:46.512682    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d309825e2abc9cd62b6a566bef818ade7733278d160f0f9fe7b79695dfa77d9\": container with ID starting with 0d309825e2abc9cd62b6a566bef818ade7733278d160f0f9fe7b79695dfa77d9 not found: ID does not exist" containerID="0d309825e2abc9cd62b6a566bef818ade7733278d160f0f9fe7b79695dfa77d9"
Oct 31 02:16:46.512811 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:46.512713    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d309825e2abc9cd62b6a566bef818ade7733278d160f0f9fe7b79695dfa77d9"} err="failed to get container status \"0d309825e2abc9cd62b6a566bef818ade7733278d160f0f9fe7b79695dfa77d9\": rpc error: code = NotFound desc = could not find container \"0d309825e2abc9cd62b6a566bef818ade7733278d160f0f9fe7b79695dfa77d9\": container with ID starting with 0d309825e2abc9cd62b6a566bef818ade7733278d160f0f9fe7b79695dfa77d9 not found: ID does not exist"
Oct 31 02:16:46.526247 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:46.526220    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw"]
Oct 31 02:16:46.532706 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:46.532675    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-20538a-f4df5f55c-brtpw"]
Oct 31 02:16:48.119659 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.119625    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd57354a-e4f0-42fc-a537-336d346ffee0" path="/var/lib/kubelet/pods/dd57354a-e4f0-42fc-a537-336d346ffee0/volumes"
Oct 31 02:16:48.652036 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.651998    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk" podUID="f7262357-a521-49d3-88f4-86e8392931b9" containerName="payload-processing" containerID="cri-o://2c221ac93e1c1f775ac1ecf2b3e962eb96969448554e703133080278cb649d4e" gracePeriod=25
Oct 31 02:16:48.675680 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.675637    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl" podUID="82a717a2-e5de-46ea-9bba-6d28baa9a9ad" containerName="payload-pre-processing" containerID="cri-o://80ca25fc89e0abcdf1bf0031fa9457ade4ec14206a36464dbd89e7b3c7b1618e" gracePeriod=25
Oct 31 02:16:48.933469 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.933091    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 02:16:48.968047 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.968008    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-workload-socket\") pod \"7f8ad70f-aced-426e-8d7a-d1c17c256582\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") "
Oct 31 02:16:48.968249 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.968091    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-envoy\") pod \"7f8ad70f-aced-426e-8d7a-d1c17c256582\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") "
Oct 31 02:16:48.968249 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.968124    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f8ad70f-aced-426e-8d7a-d1c17c256582-kube-api-access-fg9lc\" (UniqueName: \"kubernetes.io/projected/7f8ad70f-aced-426e-8d7a-d1c17c256582-kube-api-access-fg9lc\") pod \"7f8ad70f-aced-426e-8d7a-d1c17c256582\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") "
Oct 31 02:16:48.968249 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.968150    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-data\") pod \"7f8ad70f-aced-426e-8d7a-d1c17c256582\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") "
Oct 31 02:16:48.970314 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.968281    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-workload-certs\") pod \"7f8ad70f-aced-426e-8d7a-d1c17c256582\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") "
Oct 31 02:16:48.970314 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.968314    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-token\" (UniqueName: \"kubernetes.io/projected/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-token\") pod \"7f8ad70f-aced-426e-8d7a-d1c17c256582\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") "
Oct 31 02:16:48.970314 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.968401    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f8ad70f-aced-426e-8d7a-d1c17c256582-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8ad70f-aced-426e-8d7a-d1c17c256582-istiod-ca-cert\") pod \"7f8ad70f-aced-426e-8d7a-d1c17c256582\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") "
Oct 31 02:16:48.970314 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.968402    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-workload-socket" pod "7f8ad70f-aced-426e-8d7a-d1c17c256582" (UID: "7f8ad70f-aced-426e-8d7a-d1c17c256582"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:48.970314 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.968491    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-credential-socket\") pod \"7f8ad70f-aced-426e-8d7a-d1c17c256582\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") "
Oct 31 02:16:48.970314 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.968614    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-data" pod "7f8ad70f-aced-426e-8d7a-d1c17c256582" (UID: "7f8ad70f-aced-426e-8d7a-d1c17c256582"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:48.970314 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.968721    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-podinfo\") pod \"7f8ad70f-aced-426e-8d7a-d1c17c256582\" (UID: \"7f8ad70f-aced-426e-8d7a-d1c17c256582\") "
Oct 31 02:16:48.970314 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.968768    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-workload-certs" pod "7f8ad70f-aced-426e-8d7a-d1c17c256582" (UID: "7f8ad70f-aced-426e-8d7a-d1c17c256582"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:48.970314 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.968963    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-credential-socket" pod "7f8ad70f-aced-426e-8d7a-d1c17c256582" (UID: "7f8ad70f-aced-426e-8d7a-d1c17c256582"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:48.970314 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.969256    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f8ad70f-aced-426e-8d7a-d1c17c256582-istiod-ca-cert" pod "7f8ad70f-aced-426e-8d7a-d1c17c256582" (UID: "7f8ad70f-aced-426e-8d7a-d1c17c256582"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:16:48.970314 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.969437    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8ad70f-aced-426e-8d7a-d1c17c256582-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:48.970314 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.969496    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:48.970314 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.969513    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:48.970314 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.969527    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:48.970314 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.969542    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:48.972093 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.971475    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-podinfo" pod "7f8ad70f-aced-426e-8d7a-d1c17c256582" (UID: "7f8ad70f-aced-426e-8d7a-d1c17c256582"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:16:48.974913 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.974639    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f8ad70f-aced-426e-8d7a-d1c17c256582-kube-api-access-fg9lc" pod "7f8ad70f-aced-426e-8d7a-d1c17c256582" (UID: "7f8ad70f-aced-426e-8d7a-d1c17c256582"). InnerVolumeSpecName "kube-api-access-fg9lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:48.974913 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.974876    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-token" pod "7f8ad70f-aced-426e-8d7a-d1c17c256582" (UID: "7f8ad70f-aced-426e-8d7a-d1c17c256582"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:48.974913 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:48.974903    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-envoy" pod "7f8ad70f-aced-426e-8d7a-d1c17c256582" (UID: "7f8ad70f-aced-426e-8d7a-d1c17c256582"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:49.071522 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:49.071418    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:49.071522 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:49.071462    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg9lc\" (UniqueName: \"kubernetes.io/projected/7f8ad70f-aced-426e-8d7a-d1c17c256582-kube-api-access-fg9lc\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:49.071522 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:49.071481    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:49.071522 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:49.071493    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8ad70f-aced-426e-8d7a-d1c17c256582-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:49.534326 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:49.534293    5650 generic.go:356] "Generic (PLEG): container finished" podID="7f8ad70f-aced-426e-8d7a-d1c17c256582" containerID="295c7f3229683f3fc00cfd7de2ef7d1dad2e479f9e9bbdbfa89688581af56a44" exitCode=0
Oct 31 02:16:49.534827 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:49.534343    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"
Oct 31 02:16:49.534827 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:49.534384    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv" event={"ID":"7f8ad70f-aced-426e-8d7a-d1c17c256582","Type":"ContainerDied","Data":"295c7f3229683f3fc00cfd7de2ef7d1dad2e479f9e9bbdbfa89688581af56a44"}
Oct 31 02:16:49.534827 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:49.534481    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv" event={"ID":"7f8ad70f-aced-426e-8d7a-d1c17c256582","Type":"ContainerDied","Data":"c35ffc3231b773e3986a4a85b08367d812cabb126f1d1e3c3d6121cfd059f9a9"}
Oct 31 02:16:49.534827 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:49.534518    5650 scope.go:122] "RemoveContainer" containerID="295c7f3229683f3fc00cfd7de2ef7d1dad2e479f9e9bbdbfa89688581af56a44"
Oct 31 02:16:49.547328 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:49.547231    5650 scope.go:122] "RemoveContainer" containerID="b564f738c4a77bb8f71c143af141ec5494444da818473d63c951295aa1db2580"
Oct 31 02:16:49.595706 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:49.595673    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"]
Oct 31 02:16:49.600658 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:49.600624    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-sfsnv"]
Oct 31 02:16:49.605174 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:49.605144    5650 scope.go:122] "RemoveContainer" containerID="295c7f3229683f3fc00cfd7de2ef7d1dad2e479f9e9bbdbfa89688581af56a44"
Oct 31 02:16:49.605590 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:16:49.605565    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"295c7f3229683f3fc00cfd7de2ef7d1dad2e479f9e9bbdbfa89688581af56a44\": container with ID starting with 295c7f3229683f3fc00cfd7de2ef7d1dad2e479f9e9bbdbfa89688581af56a44 not found: ID does not exist" containerID="295c7f3229683f3fc00cfd7de2ef7d1dad2e479f9e9bbdbfa89688581af56a44"
Oct 31 02:16:49.605637 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:49.605602    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"295c7f3229683f3fc00cfd7de2ef7d1dad2e479f9e9bbdbfa89688581af56a44"} err="failed to get container status \"295c7f3229683f3fc00cfd7de2ef7d1dad2e479f9e9bbdbfa89688581af56a44\": rpc error: code = NotFound desc = could not find container \"295c7f3229683f3fc00cfd7de2ef7d1dad2e479f9e9bbdbfa89688581af56a44\": container with ID starting with 295c7f3229683f3fc00cfd7de2ef7d1dad2e479f9e9bbdbfa89688581af56a44 not found: ID does not exist"
Oct 31 02:16:49.605637 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:49.605624    5650 scope.go:122] "RemoveContainer" containerID="b564f738c4a77bb8f71c143af141ec5494444da818473d63c951295aa1db2580"
Oct 31 02:16:49.605947 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:16:49.605930    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b564f738c4a77bb8f71c143af141ec5494444da818473d63c951295aa1db2580\": container with ID starting with b564f738c4a77bb8f71c143af141ec5494444da818473d63c951295aa1db2580 not found: ID does not exist" containerID="b564f738c4a77bb8f71c143af141ec5494444da818473d63c951295aa1db2580"
Oct 31 02:16:49.605991 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:49.605953    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b564f738c4a77bb8f71c143af141ec5494444da818473d63c951295aa1db2580"} err="failed to get container status \"b564f738c4a77bb8f71c143af141ec5494444da818473d63c951295aa1db2580\": rpc error: code = NotFound desc = could not find container \"b564f738c4a77bb8f71c143af141ec5494444da818473d63c951295aa1db2580\": container with ID starting with b564f738c4a77bb8f71c143af141ec5494444da818473d63c951295aa1db2580 not found: ID does not exist"
Oct 31 02:16:50.123482 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:50.123429    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f8ad70f-aced-426e-8d7a-d1c17c256582" path="/var/lib/kubelet/pods/7f8ad70f-aced-426e-8d7a-d1c17c256582/volumes"
Oct 31 02:16:50.469965 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:50.463411    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl" podUID="82a717a2-e5de-46ea-9bba-6d28baa9a9ad" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:16:50.469965 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:50.464066    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk" podUID="f7262357-a521-49d3-88f4-86e8392931b9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:16:53.519927 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:53.519894    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"]
Oct 31 02:16:53.520561 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:53.520124    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww" podUID="1da683f3-ae35-4c60-a119-7711ca882f54" containerName="istio-proxy" containerID="cri-o://262ddd17bef1830147e301ee710a84267ed9f12c6375813f645e24ceef2d76b5" gracePeriod=30
Oct 31 02:16:58.613557 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:16:58.613511    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:16:58.815545 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.815483    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 02:16:58.889679 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.889642    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-workload-certs\") pod \"1da683f3-ae35-4c60-a119-7711ca882f54\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") "
Oct 31 02:16:58.889917 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.889691    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-istio-envoy\") pod \"1da683f3-ae35-4c60-a119-7711ca882f54\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") "
Oct 31 02:16:58.889917 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.889732    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1da683f3-ae35-4c60-a119-7711ca882f54-istio-token\" (UniqueName: \"kubernetes.io/projected/1da683f3-ae35-4c60-a119-7711ca882f54-istio-token\") pod \"1da683f3-ae35-4c60-a119-7711ca882f54\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") "
Oct 31 02:16:58.889917 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.889814    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-istio-data\") pod \"1da683f3-ae35-4c60-a119-7711ca882f54\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") "
Oct 31 02:16:58.890121 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.889980    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1da683f3-ae35-4c60-a119-7711ca882f54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1da683f3-ae35-4c60-a119-7711ca882f54-istio-podinfo\") pod \"1da683f3-ae35-4c60-a119-7711ca882f54\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") "
Oct 31 02:16:58.890121 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.890067    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1da683f3-ae35-4c60-a119-7711ca882f54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1da683f3-ae35-4c60-a119-7711ca882f54-istiod-ca-cert\") pod \"1da683f3-ae35-4c60-a119-7711ca882f54\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") "
Oct 31 02:16:58.890121 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.890115    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-workload-socket\") pod \"1da683f3-ae35-4c60-a119-7711ca882f54\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") "
Oct 31 02:16:58.890286 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.890203    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-credential-socket\") pod \"1da683f3-ae35-4c60-a119-7711ca882f54\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") "
Oct 31 02:16:58.890286 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.890256    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1da683f3-ae35-4c60-a119-7711ca882f54-kube-api-access-qxktp\" (UniqueName: \"kubernetes.io/projected/1da683f3-ae35-4c60-a119-7711ca882f54-kube-api-access-qxktp\") pod \"1da683f3-ae35-4c60-a119-7711ca882f54\" (UID: \"1da683f3-ae35-4c60-a119-7711ca882f54\") "
Oct 31 02:16:58.890780 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.890751    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da683f3-ae35-4c60-a119-7711ca882f54-istiod-ca-cert" pod "1da683f3-ae35-4c60-a119-7711ca882f54" (UID: "1da683f3-ae35-4c60-a119-7711ca882f54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:16:58.891322 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.891295    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1da683f3-ae35-4c60-a119-7711ca882f54-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:58.891439 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.891406    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-workload-certs" pod "1da683f3-ae35-4c60-a119-7711ca882f54" (UID: "1da683f3-ae35-4c60-a119-7711ca882f54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:58.891534 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.891513    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-credential-socket" pod "1da683f3-ae35-4c60-a119-7711ca882f54" (UID: "1da683f3-ae35-4c60-a119-7711ca882f54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:58.891607 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.891574    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-workload-socket" pod "1da683f3-ae35-4c60-a119-7711ca882f54" (UID: "1da683f3-ae35-4c60-a119-7711ca882f54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:58.891847 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.891819    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-istio-data" pod "1da683f3-ae35-4c60-a119-7711ca882f54" (UID: "1da683f3-ae35-4c60-a119-7711ca882f54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:58.893186 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.893150    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da683f3-ae35-4c60-a119-7711ca882f54-istio-token" pod "1da683f3-ae35-4c60-a119-7711ca882f54" (UID: "1da683f3-ae35-4c60-a119-7711ca882f54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:58.893436 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.893402    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-istio-envoy" pod "1da683f3-ae35-4c60-a119-7711ca882f54" (UID: "1da683f3-ae35-4c60-a119-7711ca882f54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:16:58.894022 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.894001    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da683f3-ae35-4c60-a119-7711ca882f54-kube-api-access-qxktp" pod "1da683f3-ae35-4c60-a119-7711ca882f54" (UID: "1da683f3-ae35-4c60-a119-7711ca882f54"). InnerVolumeSpecName "kube-api-access-qxktp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:16:58.895975 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.895925    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1da683f3-ae35-4c60-a119-7711ca882f54-istio-podinfo" pod "1da683f3-ae35-4c60-a119-7711ca882f54" (UID: "1da683f3-ae35-4c60-a119-7711ca882f54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:16:58.993102 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.993065    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1da683f3-ae35-4c60-a119-7711ca882f54-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:58.993102 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.993098    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:58.993102 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.993107    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1da683f3-ae35-4c60-a119-7711ca882f54-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:58.993361 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.993117    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:58.993361 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.993126    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:58.993361 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.993134    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxktp\" (UniqueName: \"kubernetes.io/projected/1da683f3-ae35-4c60-a119-7711ca882f54-kube-api-access-qxktp\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:58.993361 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.993143    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:58.993361 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:58.993151    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1da683f3-ae35-4c60-a119-7711ca882f54-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:16:59.627436 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:59.627402    5650 generic.go:356] "Generic (PLEG): container finished" podID="1da683f3-ae35-4c60-a119-7711ca882f54" containerID="262ddd17bef1830147e301ee710a84267ed9f12c6375813f645e24ceef2d76b5" exitCode=0
Oct 31 02:16:59.627946 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:59.627464    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"
Oct 31 02:16:59.627946 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:59.627517    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww" event={"ID":"1da683f3-ae35-4c60-a119-7711ca882f54","Type":"ContainerDied","Data":"262ddd17bef1830147e301ee710a84267ed9f12c6375813f645e24ceef2d76b5"}
Oct 31 02:16:59.627946 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:59.627586    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww" event={"ID":"1da683f3-ae35-4c60-a119-7711ca882f54","Type":"ContainerDied","Data":"0e3bcac15db3a3626861a2fca287f88da15f08f514be710ef24118c87ab2d84e"}
Oct 31 02:16:59.627946 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:59.627624    5650 scope.go:122] "RemoveContainer" containerID="262ddd17bef1830147e301ee710a84267ed9f12c6375813f645e24ceef2d76b5"
Oct 31 02:16:59.640104 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:59.640085    5650 scope.go:122] "RemoveContainer" containerID="06dcb568530b690aec267324abd0782c765b29d05cd80cab6a092952e8877d97"
Oct 31 02:16:59.651486 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:59.651431    5650 scope.go:122] "RemoveContainer" containerID="262ddd17bef1830147e301ee710a84267ed9f12c6375813f645e24ceef2d76b5"
Oct 31 02:16:59.651814 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:16:59.651785    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"262ddd17bef1830147e301ee710a84267ed9f12c6375813f645e24ceef2d76b5\": container with ID starting with 262ddd17bef1830147e301ee710a84267ed9f12c6375813f645e24ceef2d76b5 not found: ID does not exist" containerID="262ddd17bef1830147e301ee710a84267ed9f12c6375813f645e24ceef2d76b5"
Oct 31 02:16:59.651921 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:59.651827    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"262ddd17bef1830147e301ee710a84267ed9f12c6375813f645e24ceef2d76b5"} err="failed to get container status \"262ddd17bef1830147e301ee710a84267ed9f12c6375813f645e24ceef2d76b5\": rpc error: code = NotFound desc = could not find container \"262ddd17bef1830147e301ee710a84267ed9f12c6375813f645e24ceef2d76b5\": container with ID starting with 262ddd17bef1830147e301ee710a84267ed9f12c6375813f645e24ceef2d76b5 not found: ID does not exist"
Oct 31 02:16:59.651921 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:59.651854    5650 scope.go:122] "RemoveContainer" containerID="06dcb568530b690aec267324abd0782c765b29d05cd80cab6a092952e8877d97"
Oct 31 02:16:59.652172 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:16:59.652152    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06dcb568530b690aec267324abd0782c765b29d05cd80cab6a092952e8877d97\": container with ID starting with 06dcb568530b690aec267324abd0782c765b29d05cd80cab6a092952e8877d97 not found: ID does not exist" containerID="06dcb568530b690aec267324abd0782c765b29d05cd80cab6a092952e8877d97"
Oct 31 02:16:59.652236 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:59.652184    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06dcb568530b690aec267324abd0782c765b29d05cd80cab6a092952e8877d97"} err="failed to get container status \"06dcb568530b690aec267324abd0782c765b29d05cd80cab6a092952e8877d97\": rpc error: code = NotFound desc = could not find container \"06dcb568530b690aec267324abd0782c765b29d05cd80cab6a092952e8877d97\": container with ID starting with 06dcb568530b690aec267324abd0782c765b29d05cd80cab6a092952e8877d97 not found: ID does not exist"
Oct 31 02:16:59.655653 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:59.655631    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"]
Oct 31 02:16:59.659214 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:16:59.659192    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-20538a-openshift-default-55cc8fbd89-w5vww"]
Oct 31 02:17:00.130994 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:00.130953    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1da683f3-ae35-4c60-a119-7711ca882f54" path="/var/lib/kubelet/pods/1da683f3-ae35-4c60-a119-7711ca882f54/volumes"
Oct 31 02:17:00.461906 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:00.461861    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk" podUID="f7262357-a521-49d3-88f4-86e8392931b9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:17:00.462104 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:00.461879    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl" podUID="82a717a2-e5de-46ea-9bba-6d28baa9a9ad" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:17:04.283950 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:04.283910    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd"]
Oct 31 02:17:04.298632 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:04.298596    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c"]
Oct 31 02:17:08.195703 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.195666    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"]
Oct 31 02:17:08.196578 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.195887    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9" podUID="f8b0a988-920f-4c28-b813-7ea3c2e108ca" containerName="istio-proxy" containerID="cri-o://ab00ada38fd686ef62012a9cd8d625bf2d2a6c45a5c00510cfd496db0fb265d6" gracePeriod=30
Oct 31 02:17:08.196691 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.196618    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"]
Oct 31 02:17:08.197254 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.196791    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw" podUID="cbf0e856-3662-486c-94d1-c13cc57b633a" containerName="istio-proxy" containerID="cri-o://518442071a6687e615ccc4b3c0da2c8852311c1918f83b9bb2277ca222cc8cb4" gracePeriod=30
Oct 31 02:17:08.773075 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.773031    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468"]
Oct 31 02:17:08.773834 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.773817    5650 cpu_manager.go:414] "containerMap: removing container" podUID="87ac2c5d-5b80-44ce-904b-713134c4b4f6" containerName="cleanup"
Oct 31 02:17:08.773893 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.773836    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ac2c5d-5b80-44ce-904b-713134c4b4f6" containerName="cleanup"
Oct 31 02:17:08.773893 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.773859    5650 cpu_manager.go:414] "containerMap: removing container" podUID="283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5" containerName="revoke-keys"
Oct 31 02:17:08.773893 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.773864    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5" containerName="revoke-keys"
Oct 31 02:17:08.773893 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.773880    5650 cpu_manager.go:414] "containerMap: removing container" podUID="dd57354a-e4f0-42fc-a537-336d346ffee0" containerName="maas-api"
Oct 31 02:17:08.773893 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.773885    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd57354a-e4f0-42fc-a537-336d346ffee0" containerName="maas-api"
Oct 31 02:17:08.773893 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.773894    5650 cpu_manager.go:414] "containerMap: removing container" podUID="dfceee3a-9e84-4888-a7ff-bb5fe5b632e5" containerName="cleanup"
Oct 31 02:17:08.774100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.773899    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfceee3a-9e84-4888-a7ff-bb5fe5b632e5" containerName="cleanup"
Oct 31 02:17:08.774100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.773908    5650 cpu_manager.go:414] "containerMap: removing container" podUID="1da683f3-ae35-4c60-a119-7711ca882f54" containerName="istio-proxy"
Oct 31 02:17:08.774100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.773913    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1da683f3-ae35-4c60-a119-7711ca882f54" containerName="istio-proxy"
Oct 31 02:17:08.774100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.773928    5650 cpu_manager.go:414] "containerMap: removing container" podUID="7f8ad70f-aced-426e-8d7a-d1c17c256582" containerName="istio-proxy"
Oct 31 02:17:08.774100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.773932    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f8ad70f-aced-426e-8d7a-d1c17c256582" containerName="istio-proxy"
Oct 31 02:17:08.774100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.773940    5650 cpu_manager.go:414] "containerMap: removing container" podUID="1da683f3-ae35-4c60-a119-7711ca882f54" containerName="istio-proxy"
Oct 31 02:17:08.774100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.773944    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1da683f3-ae35-4c60-a119-7711ca882f54" containerName="istio-proxy"
Oct 31 02:17:08.774100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.773954    5650 cpu_manager.go:414] "containerMap: removing container" podUID="f6566d82-f8bc-44f8-b4ff-21edc8e67810" containerName="cleanup"
Oct 31 02:17:08.774100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.773959    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6566d82-f8bc-44f8-b4ff-21edc8e67810" containerName="cleanup"
Oct 31 02:17:08.774100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.773966    5650 cpu_manager.go:414] "containerMap: removing container" podUID="e0ae7217-5d7c-4dd3-b830-7e669721ec67" containerName="cleanup"
Oct 31 02:17:08.774100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.773972    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ae7217-5d7c-4dd3-b830-7e669721ec67" containerName="cleanup"
Oct 31 02:17:08.774100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.773980    5650 cpu_manager.go:414] "containerMap: removing container" podUID="ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf" containerName="cleanup"
Oct 31 02:17:08.774100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.773984    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf" containerName="cleanup"
Oct 31 02:17:08.774100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.773992    5650 cpu_manager.go:414] "containerMap: removing container" podUID="6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d" containerName="cleanup"
Oct 31 02:17:08.774100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.773997    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d" containerName="cleanup"
Oct 31 02:17:08.774100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.774010    5650 cpu_manager.go:414] "containerMap: removing container" podUID="7f8ad70f-aced-426e-8d7a-d1c17c256582" containerName="istio-proxy"
Oct 31 02:17:08.774100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.774014    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f8ad70f-aced-426e-8d7a-d1c17c256582" containerName="istio-proxy"
Oct 31 02:17:08.774100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.774023    5650 cpu_manager.go:414] "containerMap: removing container" podUID="9c3ce5c1-cd13-429e-9734-916e37edfb8a" containerName="cleanup"
Oct 31 02:17:08.774100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.774028    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c3ce5c1-cd13-429e-9734-916e37edfb8a" containerName="cleanup"
Oct 31 02:17:08.774810 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.774123    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5" containerName="revoke-keys"
Oct 31 02:17:08.774810 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.774132    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c3ce5c1-cd13-429e-9734-916e37edfb8a" containerName="cleanup"
Oct 31 02:17:08.774810 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.774138    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd57354a-e4f0-42fc-a537-336d346ffee0" containerName="maas-api"
Oct 31 02:17:08.774810 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.774145    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="1da683f3-ae35-4c60-a119-7711ca882f54" containerName="istio-proxy"
Oct 31 02:17:08.774810 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.774153    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="495060eb-5f49-4f96-bfe7-b89ed0c8b55e" containerName="cleanup"
Oct 31 02:17:08.774810 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.774161    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f8ad70f-aced-426e-8d7a-d1c17c256582" containerName="istio-proxy"
Oct 31 02:17:08.774810 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.774166    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="1da683f3-ae35-4c60-a119-7711ca882f54" containerName="istio-proxy"
Oct 31 02:17:08.774810 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.774173    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ae7217-5d7c-4dd3-b830-7e669721ec67" containerName="cleanup"
Oct 31 02:17:08.774810 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.774179    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef5e1b12-8ae4-4e44-a876-0e7ad00d19cf" containerName="cleanup"
Oct 31 02:17:08.774810 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.774186    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7b3b4d-d73c-4eb5-8590-47e7c9b71b9d" containerName="cleanup"
Oct 31 02:17:08.774810 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.774195    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c8cd351-2df1-4f99-a782-e63b9aab297f" containerName="cleanup"
Oct 31 02:17:08.777776 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.777755    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468"
Oct 31 02:17:08.789909 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.789880    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468"]
Oct 31 02:17:08.860799 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.860763    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:17:08.860988 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.860879    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:17:08.924525 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.924383    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5964fe0d-45cd-4bd3-9a71-0385a05f3c2c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468\" (UID: \"5964fe0d-45cd-4bd3-9a71-0385a05f3c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468"
Oct 31 02:17:08.924525 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.924473    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5964fe0d-45cd-4bd3-9a71-0385a05f3c2c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468\" (UID: \"5964fe0d-45cd-4bd3-9a71-0385a05f3c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468"
Oct 31 02:17:08.924525 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:08.924523    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwdrz\" (UniqueName: \"kubernetes.io/projected/5964fe0d-45cd-4bd3-9a71-0385a05f3c2c-kube-api-access-bwdrz\") pod \"maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468\" (UID: \"5964fe0d-45cd-4bd3-9a71-0385a05f3c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468"
Oct 31 02:17:09.013390 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.013359    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk"
Oct 31 02:17:09.025786 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.025754    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5964fe0d-45cd-4bd3-9a71-0385a05f3c2c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468\" (UID: \"5964fe0d-45cd-4bd3-9a71-0385a05f3c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468"
Oct 31 02:17:09.025942 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.025797    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5964fe0d-45cd-4bd3-9a71-0385a05f3c2c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468\" (UID: \"5964fe0d-45cd-4bd3-9a71-0385a05f3c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468"
Oct 31 02:17:09.025942 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.025843    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwdrz\" (UniqueName: \"kubernetes.io/projected/5964fe0d-45cd-4bd3-9a71-0385a05f3c2c-kube-api-access-bwdrz\") pod \"maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468\" (UID: \"5964fe0d-45cd-4bd3-9a71-0385a05f3c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468"
Oct 31 02:17:09.026578 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.026551    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5964fe0d-45cd-4bd3-9a71-0385a05f3c2c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468\" (UID: \"5964fe0d-45cd-4bd3-9a71-0385a05f3c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468"
Oct 31 02:17:09.028265 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.028241    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5964fe0d-45cd-4bd3-9a71-0385a05f3c2c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468\" (UID: \"5964fe0d-45cd-4bd3-9a71-0385a05f3c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468"
Oct 31 02:17:09.041717 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.041681    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwdrz\" (UniqueName: \"kubernetes.io/projected/5964fe0d-45cd-4bd3-9a71-0385a05f3c2c-kube-api-access-bwdrz\") pod \"maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468\" (UID: \"5964fe0d-45cd-4bd3-9a71-0385a05f3c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468"
Oct 31 02:17:09.047816 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.047786    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl"
Oct 31 02:17:09.117085 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.117049    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468"
Oct 31 02:17:09.127055 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.127020    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7262357-a521-49d3-88f4-86e8392931b9-kube-api-access-w7lr6\" (UniqueName: \"kubernetes.io/projected/f7262357-a521-49d3-88f4-86e8392931b9-kube-api-access-w7lr6\") pod \"f7262357-a521-49d3-88f4-86e8392931b9\" (UID: \"f7262357-a521-49d3-88f4-86e8392931b9\") "
Oct 31 02:17:09.127660 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.127250    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7262357-a521-49d3-88f4-86e8392931b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7262357-a521-49d3-88f4-86e8392931b9-plugins-config-volume\") pod \"f7262357-a521-49d3-88f4-86e8392931b9\" (UID: \"f7262357-a521-49d3-88f4-86e8392931b9\") "
Oct 31 02:17:09.127946 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.127913    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7262357-a521-49d3-88f4-86e8392931b9-plugins-config-volume" pod "f7262357-a521-49d3-88f4-86e8392931b9" (UID: "f7262357-a521-49d3-88f4-86e8392931b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:09.129831 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.129805    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7262357-a521-49d3-88f4-86e8392931b9-kube-api-access-w7lr6" pod "f7262357-a521-49d3-88f4-86e8392931b9" (UID: "f7262357-a521-49d3-88f4-86e8392931b9"). InnerVolumeSpecName "kube-api-access-w7lr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:09.229257 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.229224    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82a717a2-e5de-46ea-9bba-6d28baa9a9ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82a717a2-e5de-46ea-9bba-6d28baa9a9ad-plugins-config-volume\") pod \"82a717a2-e5de-46ea-9bba-6d28baa9a9ad\" (UID: \"82a717a2-e5de-46ea-9bba-6d28baa9a9ad\") "
Oct 31 02:17:09.229626 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.229510    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82a717a2-e5de-46ea-9bba-6d28baa9a9ad-kube-api-access-hx685\" (UniqueName: \"kubernetes.io/projected/82a717a2-e5de-46ea-9bba-6d28baa9a9ad-kube-api-access-hx685\") pod \"82a717a2-e5de-46ea-9bba-6d28baa9a9ad\" (UID: \"82a717a2-e5de-46ea-9bba-6d28baa9a9ad\") "
Oct 31 02:17:09.230412 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.230390    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7262357-a521-49d3-88f4-86e8392931b9-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:09.230500 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.230418    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7lr6\" (UniqueName: \"kubernetes.io/projected/f7262357-a521-49d3-88f4-86e8392931b9-kube-api-access-w7lr6\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:09.231074 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.231045    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a717a2-e5de-46ea-9bba-6d28baa9a9ad-plugins-config-volume" pod "82a717a2-e5de-46ea-9bba-6d28baa9a9ad" (UID: "82a717a2-e5de-46ea-9bba-6d28baa9a9ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:09.234304 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.234274    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a717a2-e5de-46ea-9bba-6d28baa9a9ad-kube-api-access-hx685" pod "82a717a2-e5de-46ea-9bba-6d28baa9a9ad" (UID: "82a717a2-e5de-46ea-9bba-6d28baa9a9ad"). InnerVolumeSpecName "kube-api-access-hx685". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:09.285210 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.285132    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd" podUID="f6eae8ee-0ae3-4beb-9777-e63e8853a9cb" containerName="payload-processing" containerID="cri-o://c56ab6085f2b62b923413d5d1d4c09ead0d01ea13ce894601c817df50cf82034" gracePeriod=25
Oct 31 02:17:09.299734 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.299690    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c" podUID="822c5afc-7c59-422d-a24d-d1b682127173" containerName="payload-pre-processing" containerID="cri-o://e78e05002c6d23ce58d1cfb9c3e9757eb37eb4d09fa06648ad74b19993b31276" gracePeriod=25
Oct 31 02:17:09.331400 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.331360    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82a717a2-e5de-46ea-9bba-6d28baa9a9ad-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:09.331400 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.331395    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx685\" (UniqueName: \"kubernetes.io/projected/82a717a2-e5de-46ea-9bba-6d28baa9a9ad-kube-api-access-hx685\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:09.410680 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:17:09.410637    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5964fe0d_45cd_4bd3_9a71_0385a05f3c2c.slice/crio-1b45c6d17324ea9212ba9344a37a967c9c31ce37e172d7d78362e6450c66fa9f WatchSource:0}: Error finding container 1b45c6d17324ea9212ba9344a37a967c9c31ce37e172d7d78362e6450c66fa9f: Status 404 returned error can't find the container with id 1b45c6d17324ea9212ba9344a37a967c9c31ce37e172d7d78362e6450c66fa9f
Oct 31 02:17:09.410680 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.410645    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468"]
Oct 31 02:17:09.735902 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.735864    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468" event={"ID":"5964fe0d-45cd-4bd3-9a71-0385a05f3c2c","Type":"ContainerStarted","Data":"154db362e9924ce977e3be3bcc94cde6d9537b41df921c7ab4b00366c57f76ff"}
Oct 31 02:17:09.736095 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.735941    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468" event={"ID":"5964fe0d-45cd-4bd3-9a71-0385a05f3c2c","Type":"ContainerStarted","Data":"1b45c6d17324ea9212ba9344a37a967c9c31ce37e172d7d78362e6450c66fa9f"}
Oct 31 02:17:09.737492 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.737418    5650 generic.go:356] "Generic (PLEG): container finished" podID="82a717a2-e5de-46ea-9bba-6d28baa9a9ad" containerID="80ca25fc89e0abcdf1bf0031fa9457ade4ec14206a36464dbd89e7b3c7b1618e" exitCode=0
Oct 31 02:17:09.737492 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.737483    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl"
Oct 31 02:17:09.737705 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.737515    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl" event={"ID":"82a717a2-e5de-46ea-9bba-6d28baa9a9ad","Type":"ContainerDied","Data":"80ca25fc89e0abcdf1bf0031fa9457ade4ec14206a36464dbd89e7b3c7b1618e"}
Oct 31 02:17:09.737705 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.737582    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl" event={"ID":"82a717a2-e5de-46ea-9bba-6d28baa9a9ad","Type":"ContainerDied","Data":"9cffc2ed0d3a197eb5b9cd21a42429e3020048f354f7b98d3267f13e936fe8e5"}
Oct 31 02:17:09.737820 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.737597    5650 scope.go:122] "RemoveContainer" containerID="80ca25fc89e0abcdf1bf0031fa9457ade4ec14206a36464dbd89e7b3c7b1618e"
Oct 31 02:17:09.740176 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.740149    5650 generic.go:356] "Generic (PLEG): container finished" podID="f7262357-a521-49d3-88f4-86e8392931b9" containerID="2c221ac93e1c1f775ac1ecf2b3e962eb96969448554e703133080278cb649d4e" exitCode=0
Oct 31 02:17:09.740294 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.740178    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk" event={"ID":"f7262357-a521-49d3-88f4-86e8392931b9","Type":"ContainerDied","Data":"2c221ac93e1c1f775ac1ecf2b3e962eb96969448554e703133080278cb649d4e"}
Oct 31 02:17:09.740294 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.740202    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk"
Oct 31 02:17:09.740294 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.740240    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk" event={"ID":"f7262357-a521-49d3-88f4-86e8392931b9","Type":"ContainerDied","Data":"653853d5250e02030a7722a9c589a56277460039e5a64ad9037c9153770d4ef1"}
Oct 31 02:17:09.750825 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.750804    5650 scope.go:122] "RemoveContainer" containerID="80ca25fc89e0abcdf1bf0031fa9457ade4ec14206a36464dbd89e7b3c7b1618e"
Oct 31 02:17:09.751139 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:17:09.751118    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80ca25fc89e0abcdf1bf0031fa9457ade4ec14206a36464dbd89e7b3c7b1618e\": container with ID starting with 80ca25fc89e0abcdf1bf0031fa9457ade4ec14206a36464dbd89e7b3c7b1618e not found: ID does not exist" containerID="80ca25fc89e0abcdf1bf0031fa9457ade4ec14206a36464dbd89e7b3c7b1618e"
Oct 31 02:17:09.751190 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.751147    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80ca25fc89e0abcdf1bf0031fa9457ade4ec14206a36464dbd89e7b3c7b1618e"} err="failed to get container status \"80ca25fc89e0abcdf1bf0031fa9457ade4ec14206a36464dbd89e7b3c7b1618e\": rpc error: code = NotFound desc = could not find container \"80ca25fc89e0abcdf1bf0031fa9457ade4ec14206a36464dbd89e7b3c7b1618e\": container with ID starting with 80ca25fc89e0abcdf1bf0031fa9457ade4ec14206a36464dbd89e7b3c7b1618e not found: ID does not exist"
Oct 31 02:17:09.751190 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.751165    5650 scope.go:122] "RemoveContainer" containerID="2c221ac93e1c1f775ac1ecf2b3e962eb96969448554e703133080278cb649d4e"
Oct 31 02:17:09.761001 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.760942    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468" podStartSLOduration=1.76092518 podStartE2EDuration="1.76092518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:17:09.75687522 +0000 UTC m=+3385.760589511" watchObservedRunningTime="2026-10-31 02:17:09.76092518 +0000 UTC m=+3385.764639473"
Oct 31 02:17:09.764230 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.764211    5650 scope.go:122] "RemoveContainer" containerID="2c221ac93e1c1f775ac1ecf2b3e962eb96969448554e703133080278cb649d4e"
Oct 31 02:17:09.764551 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:17:09.764531    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c221ac93e1c1f775ac1ecf2b3e962eb96969448554e703133080278cb649d4e\": container with ID starting with 2c221ac93e1c1f775ac1ecf2b3e962eb96969448554e703133080278cb649d4e not found: ID does not exist" containerID="2c221ac93e1c1f775ac1ecf2b3e962eb96969448554e703133080278cb649d4e"
Oct 31 02:17:09.764632 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.764557    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c221ac93e1c1f775ac1ecf2b3e962eb96969448554e703133080278cb649d4e"} err="failed to get container status \"2c221ac93e1c1f775ac1ecf2b3e962eb96969448554e703133080278cb649d4e\": rpc error: code = NotFound desc = could not find container \"2c221ac93e1c1f775ac1ecf2b3e962eb96969448554e703133080278cb649d4e\": container with ID starting with 2c221ac93e1c1f775ac1ecf2b3e962eb96969448554e703133080278cb649d4e not found: ID does not exist"
Oct 31 02:17:09.775965 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.775914    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl"]
Oct 31 02:17:09.780403 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.780378    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-20538a-66d844fc55-nw6fl"]
Oct 31 02:17:09.796138 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.796098    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk"]
Oct 31 02:17:09.799891 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:09.799863    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-20538a-854c797489-sc8dk"]
Oct 31 02:17:10.119386 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:10.119301    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82a717a2-e5de-46ea-9bba-6d28baa9a9ad" path="/var/lib/kubelet/pods/82a717a2-e5de-46ea-9bba-6d28baa9a9ad/volumes"
Oct 31 02:17:10.119852 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:10.119836    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7262357-a521-49d3-88f4-86e8392931b9" path="/var/lib/kubelet/pods/f7262357-a521-49d3-88f4-86e8392931b9/volumes"
Oct 31 02:17:10.752302 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:10.752266    5650 generic.go:356] "Generic (PLEG): container finished" podID="5964fe0d-45cd-4bd3-9a71-0385a05f3c2c" containerID="154db362e9924ce977e3be3bcc94cde6d9537b41df921c7ab4b00366c57f76ff" exitCode=0
Oct 31 02:17:10.752773 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:10.752328    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468" event={"ID":"5964fe0d-45cd-4bd3-9a71-0385a05f3c2c","Type":"ContainerDied","Data":"154db362e9924ce977e3be3bcc94cde6d9537b41df921c7ab4b00366c57f76ff"}
Oct 31 02:17:11.938514 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:11.938485    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468"
Oct 31 02:17:11.988798 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:11.988761    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5964fe0d-45cd-4bd3-9a71-0385a05f3c2c-kube-api-access-bwdrz\" (UniqueName: \"kubernetes.io/projected/5964fe0d-45cd-4bd3-9a71-0385a05f3c2c-kube-api-access-bwdrz\") pod \"5964fe0d-45cd-4bd3-9a71-0385a05f3c2c\" (UID: \"5964fe0d-45cd-4bd3-9a71-0385a05f3c2c\") "
Oct 31 02:17:11.988996 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:11.988833    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5964fe0d-45cd-4bd3-9a71-0385a05f3c2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5964fe0d-45cd-4bd3-9a71-0385a05f3c2c-maas-api-openshift-ca\") pod \"5964fe0d-45cd-4bd3-9a71-0385a05f3c2c\" (UID: \"5964fe0d-45cd-4bd3-9a71-0385a05f3c2c\") "
Oct 31 02:17:11.988996 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:11.988967    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5964fe0d-45cd-4bd3-9a71-0385a05f3c2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5964fe0d-45cd-4bd3-9a71-0385a05f3c2c-maas-api-xks-ca\") pod \"5964fe0d-45cd-4bd3-9a71-0385a05f3c2c\" (UID: \"5964fe0d-45cd-4bd3-9a71-0385a05f3c2c\") "
Oct 31 02:17:11.989264 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:11.989227    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5964fe0d-45cd-4bd3-9a71-0385a05f3c2c-maas-api-openshift-ca" pod "5964fe0d-45cd-4bd3-9a71-0385a05f3c2c" (UID: "5964fe0d-45cd-4bd3-9a71-0385a05f3c2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:11.989652 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:11.989595    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5964fe0d-45cd-4bd3-9a71-0385a05f3c2c-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:11.991177 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:11.991147    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5964fe0d-45cd-4bd3-9a71-0385a05f3c2c-kube-api-access-bwdrz" pod "5964fe0d-45cd-4bd3-9a71-0385a05f3c2c" (UID: "5964fe0d-45cd-4bd3-9a71-0385a05f3c2c"). InnerVolumeSpecName "kube-api-access-bwdrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:11.991528 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:11.991512    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5964fe0d-45cd-4bd3-9a71-0385a05f3c2c-maas-api-xks-ca" pod "5964fe0d-45cd-4bd3-9a71-0385a05f3c2c" (UID: "5964fe0d-45cd-4bd3-9a71-0385a05f3c2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:17:12.090792 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:12.090710    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwdrz\" (UniqueName: \"kubernetes.io/projected/5964fe0d-45cd-4bd3-9a71-0385a05f3c2c-kube-api-access-bwdrz\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:12.090792 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:12.090741    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5964fe0d-45cd-4bd3-9a71-0385a05f3c2c-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:12.769145 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:12.769116    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468"
Oct 31 02:17:12.769332 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:12.769151    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468" event={"ID":"5964fe0d-45cd-4bd3-9a71-0385a05f3c2c","Type":"ContainerDied","Data":"1b45c6d17324ea9212ba9344a37a967c9c31ce37e172d7d78362e6450c66fa9f"}
Oct 31 02:17:12.769332 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:12.769206    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b45c6d17324ea9212ba9344a37a967c9c31ce37e172d7d78362e6450c66fa9f"
Oct 31 02:17:13.529504 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.529469    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:17:13.557625 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.557568    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:17:13.625752 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.625717    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-credential-socket\") pod \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") "
Oct 31 02:17:13.625752 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.625766    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istiod-ca-cert\") pod \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") "
Oct 31 02:17:13.626038 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.625804    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbf0e856-3662-486c-94d1-c13cc57b633a-kube-api-access-czh9g\" (UniqueName: \"kubernetes.io/projected/cbf0e856-3662-486c-94d1-c13cc57b633a-kube-api-access-czh9g\") pod \"cbf0e856-3662-486c-94d1-c13cc57b633a\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") "
Oct 31 02:17:13.626038 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.625835    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-workload-certs\") pod \"cbf0e856-3662-486c-94d1-c13cc57b633a\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") "
Oct 31 02:17:13.626038 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.625955    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-workload-socket\") pod \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") "
Oct 31 02:17:13.626038 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.625986    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-podinfo\") pod \"cbf0e856-3662-486c-94d1-c13cc57b633a\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") "
Oct 31 02:17:13.626308 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.626061    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-credential-socket" pod "f8b0a988-920f-4c28-b813-7ea3c2e108ca" (UID: "f8b0a988-920f-4c28-b813-7ea3c2e108ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:13.626308 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.626069    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-token\" (UniqueName: \"kubernetes.io/projected/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-token\") pod \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") "
Oct 31 02:17:13.626308 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.626170    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-workload-socket\") pod \"cbf0e856-3662-486c-94d1-c13cc57b633a\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") "
Oct 31 02:17:13.626308 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.626242    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-credential-socket\") pod \"cbf0e856-3662-486c-94d1-c13cc57b633a\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") "
Oct 31 02:17:13.626308 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.626235    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-workload-certs" pod "cbf0e856-3662-486c-94d1-c13cc57b633a" (UID: "cbf0e856-3662-486c-94d1-c13cc57b633a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:13.626308 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.626291    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-data\") pod \"cbf0e856-3662-486c-94d1-c13cc57b633a\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") "
Oct 31 02:17:13.626658 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.626332    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbf0e856-3662-486c-94d1-c13cc57b633a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf0e856-3662-486c-94d1-c13cc57b633a-istiod-ca-cert\") pod \"cbf0e856-3662-486c-94d1-c13cc57b633a\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") "
Oct 31 02:17:13.626658 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.626383    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-token\" (UniqueName: \"kubernetes.io/projected/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-token\") pod \"cbf0e856-3662-486c-94d1-c13cc57b633a\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") "
Oct 31 02:17:13.626658 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.626288    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-workload-socket" pod "f8b0a988-920f-4c28-b813-7ea3c2e108ca" (UID: "f8b0a988-920f-4c28-b813-7ea3c2e108ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:13.626658 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.626344    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istiod-ca-cert" pod "f8b0a988-920f-4c28-b813-7ea3c2e108ca" (UID: "f8b0a988-920f-4c28-b813-7ea3c2e108ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:13.626658 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.626509    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-workload-certs\") pod \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") "
Oct 31 02:17:13.626658 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.626556    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-envoy\") pod \"cbf0e856-3662-486c-94d1-c13cc57b633a\" (UID: \"cbf0e856-3662-486c-94d1-c13cc57b633a\") "
Oct 31 02:17:13.626658 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.626594    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8b0a988-920f-4c28-b813-7ea3c2e108ca-kube-api-access-wlx8t\" (UniqueName: \"kubernetes.io/projected/f8b0a988-920f-4c28-b813-7ea3c2e108ca-kube-api-access-wlx8t\") pod \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") "
Oct 31 02:17:13.626658 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.626630    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-podinfo\") pod \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") "
Oct 31 02:17:13.627147 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.626704    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-envoy\") pod \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") "
Oct 31 02:17:13.627147 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.626749    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-data\") pod \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\" (UID: \"f8b0a988-920f-4c28-b813-7ea3c2e108ca\") "
Oct 31 02:17:13.627147 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.626507    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-workload-socket" pod "cbf0e856-3662-486c-94d1-c13cc57b633a" (UID: "cbf0e856-3662-486c-94d1-c13cc57b633a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:13.627147 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.626568    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-credential-socket" pod "cbf0e856-3662-486c-94d1-c13cc57b633a" (UID: "cbf0e856-3662-486c-94d1-c13cc57b633a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:13.627147 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.626746    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-workload-certs" pod "f8b0a988-920f-4c28-b813-7ea3c2e108ca" (UID: "f8b0a988-920f-4c28-b813-7ea3c2e108ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:13.627147 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.626811    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbf0e856-3662-486c-94d1-c13cc57b633a-istiod-ca-cert" pod "cbf0e856-3662-486c-94d1-c13cc57b633a" (UID: "cbf0e856-3662-486c-94d1-c13cc57b633a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:13.627563 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.627180    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-data" pod "cbf0e856-3662-486c-94d1-c13cc57b633a" (UID: "cbf0e856-3662-486c-94d1-c13cc57b633a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:13.627563 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.627406    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-data" pod "f8b0a988-920f-4c28-b813-7ea3c2e108ca" (UID: "f8b0a988-920f-4c28-b813-7ea3c2e108ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:13.627563 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.627469    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.627563 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.627494    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.627563 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.627509    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.627563 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.627523    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbf0e856-3662-486c-94d1-c13cc57b633a-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.627563 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.627538    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.627563 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.627552    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.628077 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.627566    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.628077 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.627583    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.628077 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.627597    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.628922 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.628891    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-token" pod "f8b0a988-920f-4c28-b813-7ea3c2e108ca" (UID: "f8b0a988-920f-4c28-b813-7ea3c2e108ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:13.630244 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.630224    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-podinfo" pod "f8b0a988-920f-4c28-b813-7ea3c2e108ca" (UID: "f8b0a988-920f-4c28-b813-7ea3c2e108ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:17:13.630603 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.630567    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-envoy" pod "f8b0a988-920f-4c28-b813-7ea3c2e108ca" (UID: "f8b0a988-920f-4c28-b813-7ea3c2e108ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:13.631031 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.631008    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-token" pod "cbf0e856-3662-486c-94d1-c13cc57b633a" (UID: "cbf0e856-3662-486c-94d1-c13cc57b633a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:13.631031 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.631021    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-podinfo" pod "cbf0e856-3662-486c-94d1-c13cc57b633a" (UID: "cbf0e856-3662-486c-94d1-c13cc57b633a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:17:13.631192 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.631043    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf0e856-3662-486c-94d1-c13cc57b633a-kube-api-access-czh9g" pod "cbf0e856-3662-486c-94d1-c13cc57b633a" (UID: "cbf0e856-3662-486c-94d1-c13cc57b633a"). InnerVolumeSpecName "kube-api-access-czh9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:13.631261 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.631186    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b0a988-920f-4c28-b813-7ea3c2e108ca-kube-api-access-wlx8t" pod "f8b0a988-920f-4c28-b813-7ea3c2e108ca" (UID: "f8b0a988-920f-4c28-b813-7ea3c2e108ca"). InnerVolumeSpecName "kube-api-access-wlx8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:13.632364 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.632339    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-envoy" pod "cbf0e856-3662-486c-94d1-c13cc57b633a" (UID: "cbf0e856-3662-486c-94d1-c13cc57b633a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:13.729182 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.729141    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.729182 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.729183    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlx8t\" (UniqueName: \"kubernetes.io/projected/f8b0a988-920f-4c28-b813-7ea3c2e108ca-kube-api-access-wlx8t\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.729182 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.729195    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.729478 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.729204    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.729478 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.729212    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.729478 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.729221    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czh9g\" (UniqueName: \"kubernetes.io/projected/cbf0e856-3662-486c-94d1-c13cc57b633a-kube-api-access-czh9g\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.729478 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.729231    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.729478 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.729246    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8b0a988-920f-4c28-b813-7ea3c2e108ca-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.729478 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.729258    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbf0e856-3662-486c-94d1-c13cc57b633a-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:13.777932 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.777838    5650 generic.go:356] "Generic (PLEG): container finished" podID="cbf0e856-3662-486c-94d1-c13cc57b633a" containerID="518442071a6687e615ccc4b3c0da2c8852311c1918f83b9bb2277ca222cc8cb4" exitCode=0
Oct 31 02:17:13.777932 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.777883    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"
Oct 31 02:17:13.778179 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.777926    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw" event={"ID":"cbf0e856-3662-486c-94d1-c13cc57b633a","Type":"ContainerDied","Data":"518442071a6687e615ccc4b3c0da2c8852311c1918f83b9bb2277ca222cc8cb4"}
Oct 31 02:17:13.778179 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.778017    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw" event={"ID":"cbf0e856-3662-486c-94d1-c13cc57b633a","Type":"ContainerDied","Data":"ca1cfd160294ce030352f95cc0471ab504980e46fe2633c199a024c4ab37d930"}
Oct 31 02:17:13.778179 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.778050    5650 scope.go:122] "RemoveContainer" containerID="518442071a6687e615ccc4b3c0da2c8852311c1918f83b9bb2277ca222cc8cb4"
Oct 31 02:17:13.779830 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.779807    5650 generic.go:356] "Generic (PLEG): container finished" podID="f8b0a988-920f-4c28-b813-7ea3c2e108ca" containerID="ab00ada38fd686ef62012a9cd8d625bf2d2a6c45a5c00510cfd496db0fb265d6" exitCode=0
Oct 31 02:17:13.779966 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.779866    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"
Oct 31 02:17:13.779966 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.779900    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9" event={"ID":"f8b0a988-920f-4c28-b813-7ea3c2e108ca","Type":"ContainerDied","Data":"ab00ada38fd686ef62012a9cd8d625bf2d2a6c45a5c00510cfd496db0fb265d6"}
Oct 31 02:17:13.780075 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.779964    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9" event={"ID":"f8b0a988-920f-4c28-b813-7ea3c2e108ca","Type":"ContainerDied","Data":"a81211c73516af0a1d299d08130a70d1d9db7acc7c8c69a4e4fd94fdbfae85d4"}
Oct 31 02:17:13.791504 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.791483    5650 scope.go:122] "RemoveContainer" containerID="518442071a6687e615ccc4b3c0da2c8852311c1918f83b9bb2277ca222cc8cb4"
Oct 31 02:17:13.791821 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:17:13.791802    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"518442071a6687e615ccc4b3c0da2c8852311c1918f83b9bb2277ca222cc8cb4\": container with ID starting with 518442071a6687e615ccc4b3c0da2c8852311c1918f83b9bb2277ca222cc8cb4 not found: ID does not exist" containerID="518442071a6687e615ccc4b3c0da2c8852311c1918f83b9bb2277ca222cc8cb4"
Oct 31 02:17:13.791878 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.791828    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518442071a6687e615ccc4b3c0da2c8852311c1918f83b9bb2277ca222cc8cb4"} err="failed to get container status \"518442071a6687e615ccc4b3c0da2c8852311c1918f83b9bb2277ca222cc8cb4\": rpc error: code = NotFound desc = could not find container \"518442071a6687e615ccc4b3c0da2c8852311c1918f83b9bb2277ca222cc8cb4\": container with ID starting with 518442071a6687e615ccc4b3c0da2c8852311c1918f83b9bb2277ca222cc8cb4 not found: ID does not exist"
Oct 31 02:17:13.791878 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.791845    5650 scope.go:122] "RemoveContainer" containerID="ab00ada38fd686ef62012a9cd8d625bf2d2a6c45a5c00510cfd496db0fb265d6"
Oct 31 02:17:13.802712 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.802687    5650 scope.go:122] "RemoveContainer" containerID="ab00ada38fd686ef62012a9cd8d625bf2d2a6c45a5c00510cfd496db0fb265d6"
Oct 31 02:17:13.803562 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:17:13.803531    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab00ada38fd686ef62012a9cd8d625bf2d2a6c45a5c00510cfd496db0fb265d6\": container with ID starting with ab00ada38fd686ef62012a9cd8d625bf2d2a6c45a5c00510cfd496db0fb265d6 not found: ID does not exist" containerID="ab00ada38fd686ef62012a9cd8d625bf2d2a6c45a5c00510cfd496db0fb265d6"
Oct 31 02:17:13.803664 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.803571    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab00ada38fd686ef62012a9cd8d625bf2d2a6c45a5c00510cfd496db0fb265d6"} err="failed to get container status \"ab00ada38fd686ef62012a9cd8d625bf2d2a6c45a5c00510cfd496db0fb265d6\": rpc error: code = NotFound desc = could not find container \"ab00ada38fd686ef62012a9cd8d625bf2d2a6c45a5c00510cfd496db0fb265d6\": container with ID starting with ab00ada38fd686ef62012a9cd8d625bf2d2a6c45a5c00510cfd496db0fb265d6 not found: ID does not exist"
Oct 31 02:17:13.807864 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.807840    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"]
Oct 31 02:17:13.811269 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.811248    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-mwchw"]
Oct 31 02:17:13.829304 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.829274    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"]
Oct 31 02:17:13.832969 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:13.832947    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-kx5k9"]
Oct 31 02:17:14.122046 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:14.121965    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbf0e856-3662-486c-94d1-c13cc57b633a" path="/var/lib/kubelet/pods/cbf0e856-3662-486c-94d1-c13cc57b633a/volumes"
Oct 31 02:17:14.122552 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:14.122532    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8b0a988-920f-4c28-b813-7ea3c2e108ca" path="/var/lib/kubelet/pods/f8b0a988-920f-4c28-b813-7ea3c2e108ca/volumes"
Oct 31 02:17:17.837844 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:17.837801    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd" podUID="f6eae8ee-0ae3-4beb-9777-e63e8853a9cb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:17:17.838284 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:17.837894    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c" podUID="822c5afc-7c59-422d-a24d-d1b682127173" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:17:18.082584 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:18.082551    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn"]
Oct 31 02:17:18.082846 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:18.082819    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn" podUID="d1b923c0-10b1-4fd5-912f-57d22e29a284" containerName="maas-api" containerID="cri-o://a94e6c4e3f09a426edeed1aec734bc4ff4abc8608389687263d70af91e73a192" gracePeriod=30
Oct 31 02:17:18.379567 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:18.379218    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn"
Oct 31 02:17:18.433144 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:18.433109    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1b923c0-10b1-4fd5-912f-57d22e29a284-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1b923c0-10b1-4fd5-912f-57d22e29a284-maas-api-tls\") pod \"d1b923c0-10b1-4fd5-912f-57d22e29a284\" (UID: \"d1b923c0-10b1-4fd5-912f-57d22e29a284\") "
Oct 31 02:17:18.433363 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:18.433346    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1b923c0-10b1-4fd5-912f-57d22e29a284-kube-api-access-fzbtm\" (UniqueName: \"kubernetes.io/projected/d1b923c0-10b1-4fd5-912f-57d22e29a284-kube-api-access-fzbtm\") pod \"d1b923c0-10b1-4fd5-912f-57d22e29a284\" (UID: \"d1b923c0-10b1-4fd5-912f-57d22e29a284\") "
Oct 31 02:17:18.433496 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:18.433411    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1b923c0-10b1-4fd5-912f-57d22e29a284-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1b923c0-10b1-4fd5-912f-57d22e29a284-metrics-certs\") pod \"d1b923c0-10b1-4fd5-912f-57d22e29a284\" (UID: \"d1b923c0-10b1-4fd5-912f-57d22e29a284\") "
Oct 31 02:17:18.433583 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:18.433491    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1b923c0-10b1-4fd5-912f-57d22e29a284-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1b923c0-10b1-4fd5-912f-57d22e29a284-openshift-service-ca\") pod \"d1b923c0-10b1-4fd5-912f-57d22e29a284\" (UID: \"d1b923c0-10b1-4fd5-912f-57d22e29a284\") "
Oct 31 02:17:18.434030 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:18.434002    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1b923c0-10b1-4fd5-912f-57d22e29a284-openshift-service-ca" pod "d1b923c0-10b1-4fd5-912f-57d22e29a284" (UID: "d1b923c0-10b1-4fd5-912f-57d22e29a284"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:18.435654 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:18.435625    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1b923c0-10b1-4fd5-912f-57d22e29a284-metrics-certs" pod "d1b923c0-10b1-4fd5-912f-57d22e29a284" (UID: "d1b923c0-10b1-4fd5-912f-57d22e29a284"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:17:18.435906 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:18.435880    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1b923c0-10b1-4fd5-912f-57d22e29a284-maas-api-tls" pod "d1b923c0-10b1-4fd5-912f-57d22e29a284" (UID: "d1b923c0-10b1-4fd5-912f-57d22e29a284"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:17:18.436000 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:18.435911    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b923c0-10b1-4fd5-912f-57d22e29a284-kube-api-access-fzbtm" pod "d1b923c0-10b1-4fd5-912f-57d22e29a284" (UID: "d1b923c0-10b1-4fd5-912f-57d22e29a284"). InnerVolumeSpecName "kube-api-access-fzbtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:18.548495 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:18.535337    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzbtm\" (UniqueName: \"kubernetes.io/projected/d1b923c0-10b1-4fd5-912f-57d22e29a284-kube-api-access-fzbtm\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:18.548495 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:18.535386    5650 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1b923c0-10b1-4fd5-912f-57d22e29a284-metrics-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:18.548495 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:18.535404    5650 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1b923c0-10b1-4fd5-912f-57d22e29a284-openshift-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:18.548495 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:18.535419    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1b923c0-10b1-4fd5-912f-57d22e29a284-maas-api-tls\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:18.819507 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:18.819433    5650 generic.go:356] "Generic (PLEG): container finished" podID="d1b923c0-10b1-4fd5-912f-57d22e29a284" containerID="a94e6c4e3f09a426edeed1aec734bc4ff4abc8608389687263d70af91e73a192" exitCode=0
Oct 31 02:17:18.819696 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:18.819512    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn" event={"ID":"d1b923c0-10b1-4fd5-912f-57d22e29a284","Type":"ContainerDied","Data":"a94e6c4e3f09a426edeed1aec734bc4ff4abc8608389687263d70af91e73a192"}
Oct 31 02:17:18.819696 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:18.819571    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn" event={"ID":"d1b923c0-10b1-4fd5-912f-57d22e29a284","Type":"ContainerDied","Data":"acb07badfc071ccfec7b8b7f771e42c0403cd0dfd7c6541efffc525303fa3807"}
Oct 31 02:17:18.819696 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:18.819571    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn"
Oct 31 02:17:18.819913 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:18.819597    5650 scope.go:122] "RemoveContainer" containerID="a94e6c4e3f09a426edeed1aec734bc4ff4abc8608389687263d70af91e73a192"
Oct 31 02:17:18.834624 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:18.834524    5650 scope.go:122] "RemoveContainer" containerID="a94e6c4e3f09a426edeed1aec734bc4ff4abc8608389687263d70af91e73a192"
Oct 31 02:17:18.835169 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:17:18.835082    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a94e6c4e3f09a426edeed1aec734bc4ff4abc8608389687263d70af91e73a192\": container with ID starting with a94e6c4e3f09a426edeed1aec734bc4ff4abc8608389687263d70af91e73a192 not found: ID does not exist" containerID="a94e6c4e3f09a426edeed1aec734bc4ff4abc8608389687263d70af91e73a192"
Oct 31 02:17:18.835169 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:18.835131    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a94e6c4e3f09a426edeed1aec734bc4ff4abc8608389687263d70af91e73a192"} err="failed to get container status \"a94e6c4e3f09a426edeed1aec734bc4ff4abc8608389687263d70af91e73a192\": rpc error: code = NotFound desc = could not find container \"a94e6c4e3f09a426edeed1aec734bc4ff4abc8608389687263d70af91e73a192\": container with ID starting with a94e6c4e3f09a426edeed1aec734bc4ff4abc8608389687263d70af91e73a192 not found: ID does not exist"
Oct 31 02:17:18.847339 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:18.847310    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn"]
Oct 31 02:17:18.853876 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:18.853848    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-28e4f3-797f5bfb6d-5vzvn"]
Oct 31 02:17:18.921629 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:18.921593    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g"]
Oct 31 02:17:20.119566 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:20.119534    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1b923c0-10b1-4fd5-912f-57d22e29a284" path="/var/lib/kubelet/pods/d1b923c0-10b1-4fd5-912f-57d22e29a284/volumes"
Oct 31 02:17:20.656180 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:20.656144    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"]
Oct 31 02:17:20.656438 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:20.656403    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q" podUID="5edd74fe-6362-4479-b949-c44183eb2b0d" containerName="istio-proxy" containerID="cri-o://bf8d7c2b0239c4d535a26dfe1ad8b4e904f3b8344339efb667213066e153bab6" gracePeriod=30
Oct 31 02:17:21.200900 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.200863    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh"]
Oct 31 02:17:21.201810 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.201791    5650 cpu_manager.go:414] "containerMap: removing container" podUID="cbf0e856-3662-486c-94d1-c13cc57b633a" containerName="istio-proxy"
Oct 31 02:17:21.201857 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.201813    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbf0e856-3662-486c-94d1-c13cc57b633a" containerName="istio-proxy"
Oct 31 02:17:21.201857 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.201850    5650 cpu_manager.go:414] "containerMap: removing container" podUID="f8b0a988-920f-4c28-b813-7ea3c2e108ca" containerName="istio-proxy"
Oct 31 02:17:21.201926 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.201858    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8b0a988-920f-4c28-b813-7ea3c2e108ca" containerName="istio-proxy"
Oct 31 02:17:21.201926 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.201868    5650 cpu_manager.go:414] "containerMap: removing container" podUID="82a717a2-e5de-46ea-9bba-6d28baa9a9ad" containerName="payload-pre-processing"
Oct 31 02:17:21.201926 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.201876    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82a717a2-e5de-46ea-9bba-6d28baa9a9ad" containerName="payload-pre-processing"
Oct 31 02:17:21.201926 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.201888    5650 cpu_manager.go:414] "containerMap: removing container" podUID="d1b923c0-10b1-4fd5-912f-57d22e29a284" containerName="maas-api"
Oct 31 02:17:21.201926 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.201894    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1b923c0-10b1-4fd5-912f-57d22e29a284" containerName="maas-api"
Oct 31 02:17:21.201926 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.201921    5650 cpu_manager.go:414] "containerMap: removing container" podUID="5964fe0d-45cd-4bd3-9a71-0385a05f3c2c" containerName="revoke-keys"
Oct 31 02:17:21.202116 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.201929    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5964fe0d-45cd-4bd3-9a71-0385a05f3c2c" containerName="revoke-keys"
Oct 31 02:17:21.202116 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.201966    5650 cpu_manager.go:414] "containerMap: removing container" podUID="f7262357-a521-49d3-88f4-86e8392931b9" containerName="payload-processing"
Oct 31 02:17:21.202116 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.201974    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7262357-a521-49d3-88f4-86e8392931b9" containerName="payload-processing"
Oct 31 02:17:21.202209 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.202129    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="5964fe0d-45cd-4bd3-9a71-0385a05f3c2c" containerName="revoke-keys"
Oct 31 02:17:21.202209 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.202148    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7262357-a521-49d3-88f4-86e8392931b9" containerName="payload-processing"
Oct 31 02:17:21.202209 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.202164    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f8ad70f-aced-426e-8d7a-d1c17c256582" containerName="istio-proxy"
Oct 31 02:17:21.202209 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.202176    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbf0e856-3662-486c-94d1-c13cc57b633a" containerName="istio-proxy"
Oct 31 02:17:21.202209 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.202185    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8b0a988-920f-4c28-b813-7ea3c2e108ca" containerName="istio-proxy"
Oct 31 02:17:21.202209 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.202190    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="82a717a2-e5de-46ea-9bba-6d28baa9a9ad" containerName="payload-pre-processing"
Oct 31 02:17:21.202209 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.202195    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1b923c0-10b1-4fd5-912f-57d22e29a284" containerName="maas-api"
Oct 31 02:17:21.207345 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.207320    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh"
Oct 31 02:17:21.213429 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.213402    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:17:21.213429 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.213481    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:17:21.219965 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.219923    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh"]
Oct 31 02:17:21.287745 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.287706    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68500858-f749-44a3-95d1-ebb88191c582-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh\" (UID: \"68500858-f749-44a3-95d1-ebb88191c582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh"
Oct 31 02:17:21.287954 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.287928    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68500858-f749-44a3-95d1-ebb88191c582-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh\" (UID: \"68500858-f749-44a3-95d1-ebb88191c582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh"
Oct 31 02:17:21.288003 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.287967    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27957\" (UniqueName: \"kubernetes.io/projected/68500858-f749-44a3-95d1-ebb88191c582-kube-api-access-27957\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh\" (UID: \"68500858-f749-44a3-95d1-ebb88191c582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh"
Oct 31 02:17:21.389594 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.389556    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68500858-f749-44a3-95d1-ebb88191c582-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh\" (UID: \"68500858-f749-44a3-95d1-ebb88191c582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh"
Oct 31 02:17:21.389594 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.389593    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27957\" (UniqueName: \"kubernetes.io/projected/68500858-f749-44a3-95d1-ebb88191c582-kube-api-access-27957\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh\" (UID: \"68500858-f749-44a3-95d1-ebb88191c582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh"
Oct 31 02:17:21.389807 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.389692    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68500858-f749-44a3-95d1-ebb88191c582-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh\" (UID: \"68500858-f749-44a3-95d1-ebb88191c582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh"
Oct 31 02:17:21.390365 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.390340    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68500858-f749-44a3-95d1-ebb88191c582-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh\" (UID: \"68500858-f749-44a3-95d1-ebb88191c582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh"
Oct 31 02:17:21.391931 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.391911    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68500858-f749-44a3-95d1-ebb88191c582-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh\" (UID: \"68500858-f749-44a3-95d1-ebb88191c582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh"
Oct 31 02:17:21.404330 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.404306    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27957\" (UniqueName: \"kubernetes.io/projected/68500858-f749-44a3-95d1-ebb88191c582-kube-api-access-27957\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh\" (UID: \"68500858-f749-44a3-95d1-ebb88191c582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh"
Oct 31 02:17:21.531812 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.530624    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh"
Oct 31 02:17:21.844877 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.844845    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh"]
Oct 31 02:17:21.856305 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:21.856276    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh" event={"ID":"68500858-f749-44a3-95d1-ebb88191c582","Type":"ContainerStarted","Data":"54a6fabbc70422e67652d1f0236fe13b3e8d09c42249e13ed9ddcf73ee0ded4c"}
Oct 31 02:17:22.887471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:22.884151    5650 generic.go:356] "Generic (PLEG): container finished" podID="68500858-f749-44a3-95d1-ebb88191c582" containerID="98badd32701c5777a130b20b59df40c1ac6532c8874a91eb897b139db898ae20" exitCode=0
Oct 31 02:17:22.887471 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:22.884500    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh" event={"ID":"68500858-f749-44a3-95d1-ebb88191c582","Type":"ContainerDied","Data":"98badd32701c5777a130b20b59df40c1ac6532c8874a91eb897b139db898ae20"}
Oct 31 02:17:23.024616 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:23.024528    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468"]
Oct 31 02:17:23.028154 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:23.028121    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-91b6b1-2n468"]
Oct 31 02:17:23.922359 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:23.922302    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g" podUID="44fe8812-7863-4904-bb34-5ee90f27db75" containerName="payload-pre-processing" containerID="cri-o://49bbaf641308eeb3b1a50644103c0959e57337e28375476c0b612858d26e3bd5" gracePeriod=25
Oct 31 02:17:24.145844 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:24.145777    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5964fe0d-45cd-4bd3-9a71-0385a05f3c2c" path="/var/lib/kubelet/pods/5964fe0d-45cd-4bd3-9a71-0385a05f3c2c/volumes"
Oct 31 02:17:24.178175 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:24.178145    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh"
Oct 31 02:17:24.254262 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:24.254224    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68500858-f749-44a3-95d1-ebb88191c582-kube-api-access-27957\" (UniqueName: \"kubernetes.io/projected/68500858-f749-44a3-95d1-ebb88191c582-kube-api-access-27957\") pod \"68500858-f749-44a3-95d1-ebb88191c582\" (UID: \"68500858-f749-44a3-95d1-ebb88191c582\") "
Oct 31 02:17:24.254508 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:24.254411    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68500858-f749-44a3-95d1-ebb88191c582-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68500858-f749-44a3-95d1-ebb88191c582-maas-api-openshift-ca\") pod \"68500858-f749-44a3-95d1-ebb88191c582\" (UID: \"68500858-f749-44a3-95d1-ebb88191c582\") "
Oct 31 02:17:24.254608 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:24.254562    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68500858-f749-44a3-95d1-ebb88191c582-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68500858-f749-44a3-95d1-ebb88191c582-maas-api-xks-ca\") pod \"68500858-f749-44a3-95d1-ebb88191c582\" (UID: \"68500858-f749-44a3-95d1-ebb88191c582\") "
Oct 31 02:17:24.254889 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:24.254859    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68500858-f749-44a3-95d1-ebb88191c582-maas-api-openshift-ca" pod "68500858-f749-44a3-95d1-ebb88191c582" (UID: "68500858-f749-44a3-95d1-ebb88191c582"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:24.256134 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:24.255755    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68500858-f749-44a3-95d1-ebb88191c582-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:24.257360 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:24.257326    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68500858-f749-44a3-95d1-ebb88191c582-kube-api-access-27957" pod "68500858-f749-44a3-95d1-ebb88191c582" (UID: "68500858-f749-44a3-95d1-ebb88191c582"). InnerVolumeSpecName "kube-api-access-27957". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:24.258923 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:24.258892    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68500858-f749-44a3-95d1-ebb88191c582-maas-api-xks-ca" pod "68500858-f749-44a3-95d1-ebb88191c582" (UID: "68500858-f749-44a3-95d1-ebb88191c582"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:17:24.374603 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:24.372003    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27957\" (UniqueName: \"kubernetes.io/projected/68500858-f749-44a3-95d1-ebb88191c582-kube-api-access-27957\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:24.374603 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:24.372050    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68500858-f749-44a3-95d1-ebb88191c582-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:24.900123 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:24.900091    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh"
Oct 31 02:17:24.900321 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:24.900129    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh" event={"ID":"68500858-f749-44a3-95d1-ebb88191c582","Type":"ContainerDied","Data":"54a6fabbc70422e67652d1f0236fe13b3e8d09c42249e13ed9ddcf73ee0ded4c"}
Oct 31 02:17:24.900321 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:24.900183    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54a6fabbc70422e67652d1f0236fe13b3e8d09c42249e13ed9ddcf73ee0ded4c"
Oct 31 02:17:25.937507 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:25.937432    5650 generic.go:356] "Generic (PLEG): container finished" podID="5edd74fe-6362-4479-b949-c44183eb2b0d" containerID="bf8d7c2b0239c4d535a26dfe1ad8b4e904f3b8344339efb667213066e153bab6" exitCode=0
Oct 31 02:17:25.937914 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:25.937526    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q" event={"ID":"5edd74fe-6362-4479-b949-c44183eb2b0d","Type":"ContainerDied","Data":"bf8d7c2b0239c4d535a26dfe1ad8b4e904f3b8344339efb667213066e153bab6"}
Oct 31 02:17:25.950016 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:25.949958    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:17:26.000334 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.000297    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-podinfo\") pod \"5edd74fe-6362-4479-b949-c44183eb2b0d\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") "
Oct 31 02:17:26.000566 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.000409    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-token\" (UniqueName: \"kubernetes.io/projected/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-token\") pod \"5edd74fe-6362-4479-b949-c44183eb2b0d\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") "
Oct 31 02:17:26.000566 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.000535    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-credential-socket\") pod \"5edd74fe-6362-4479-b949-c44183eb2b0d\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") "
Oct 31 02:17:26.000709 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.000613    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-data\") pod \"5edd74fe-6362-4479-b949-c44183eb2b0d\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") "
Oct 31 02:17:26.000709 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.000668    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-workload-socket\") pod \"5edd74fe-6362-4479-b949-c44183eb2b0d\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") "
Oct 31 02:17:26.000822 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.000749    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-workload-certs\") pod \"5edd74fe-6362-4479-b949-c44183eb2b0d\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") "
Oct 31 02:17:26.000822 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.000787    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5edd74fe-6362-4479-b949-c44183eb2b0d-kube-api-access-47znk\" (UniqueName: \"kubernetes.io/projected/5edd74fe-6362-4479-b949-c44183eb2b0d-kube-api-access-47znk\") pod \"5edd74fe-6362-4479-b949-c44183eb2b0d\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") "
Oct 31 02:17:26.000939 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.000837    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5edd74fe-6362-4479-b949-c44183eb2b0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5edd74fe-6362-4479-b949-c44183eb2b0d-istiod-ca-cert\") pod \"5edd74fe-6362-4479-b949-c44183eb2b0d\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") "
Oct 31 02:17:26.000939 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.000868    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-envoy\") pod \"5edd74fe-6362-4479-b949-c44183eb2b0d\" (UID: \"5edd74fe-6362-4479-b949-c44183eb2b0d\") "
Oct 31 02:17:26.001089 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.001062    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-workload-certs" pod "5edd74fe-6362-4479-b949-c44183eb2b0d" (UID: "5edd74fe-6362-4479-b949-c44183eb2b0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:26.001598 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.001573    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:26.001831 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.001809    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-data" pod "5edd74fe-6362-4479-b949-c44183eb2b0d" (UID: "5edd74fe-6362-4479-b949-c44183eb2b0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:26.002025 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.002005    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-workload-socket" pod "5edd74fe-6362-4479-b949-c44183eb2b0d" (UID: "5edd74fe-6362-4479-b949-c44183eb2b0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:26.002368 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.002316    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-credential-socket" pod "5edd74fe-6362-4479-b949-c44183eb2b0d" (UID: "5edd74fe-6362-4479-b949-c44183eb2b0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:26.002762 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.002733    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5edd74fe-6362-4479-b949-c44183eb2b0d-istiod-ca-cert" pod "5edd74fe-6362-4479-b949-c44183eb2b0d" (UID: "5edd74fe-6362-4479-b949-c44183eb2b0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:26.006492 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.005688    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-envoy" pod "5edd74fe-6362-4479-b949-c44183eb2b0d" (UID: "5edd74fe-6362-4479-b949-c44183eb2b0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:26.006492 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.005813    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5edd74fe-6362-4479-b949-c44183eb2b0d-kube-api-access-47znk" pod "5edd74fe-6362-4479-b949-c44183eb2b0d" (UID: "5edd74fe-6362-4479-b949-c44183eb2b0d"). InnerVolumeSpecName "kube-api-access-47znk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:26.006492 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.005901    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-podinfo" pod "5edd74fe-6362-4479-b949-c44183eb2b0d" (UID: "5edd74fe-6362-4479-b949-c44183eb2b0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:17:26.006492 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.006334    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-token" pod "5edd74fe-6362-4479-b949-c44183eb2b0d" (UID: "5edd74fe-6362-4479-b949-c44183eb2b0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:26.103535 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.103497    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:26.103535 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.103528    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:26.103535 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.103538    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47znk\" (UniqueName: \"kubernetes.io/projected/5edd74fe-6362-4479-b949-c44183eb2b0d-kube-api-access-47znk\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:26.103789 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.103547    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5edd74fe-6362-4479-b949-c44183eb2b0d-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:26.103789 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.103557    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:26.103789 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.103566    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:26.103789 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.103574    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5edd74fe-6362-4479-b949-c44183eb2b0d-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:26.103789 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.103583    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edd74fe-6362-4479-b949-c44183eb2b0d-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:26.947694 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.947663    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"
Oct 31 02:17:26.948175 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.947689    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q" event={"ID":"5edd74fe-6362-4479-b949-c44183eb2b0d","Type":"ContainerDied","Data":"7f17920f3522889f107426af5bd536dd8ed25ab0757528a956cdd36f778a003b"}
Oct 31 02:17:26.948175 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.947753    5650 scope.go:122] "RemoveContainer" containerID="bf8d7c2b0239c4d535a26dfe1ad8b4e904f3b8344339efb667213066e153bab6"
Oct 31 02:17:26.973659 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.973615    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"]
Oct 31 02:17:26.992243 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:26.992198    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-czj4q"]
Oct 31 02:17:27.837617 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:27.837574    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd" podUID="f6eae8ee-0ae3-4beb-9777-e63e8853a9cb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:17:27.837617 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:27.837585    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c" podUID="822c5afc-7c59-422d-a24d-d1b682127173" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:17:28.119249 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:28.119169    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5edd74fe-6362-4479-b949-c44183eb2b0d" path="/var/lib/kubelet/pods/5edd74fe-6362-4479-b949-c44183eb2b0d/volumes"
Oct 31 02:17:28.554175 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:17:28.554135    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:17:29.698536 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.697421    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd"
Oct 31 02:17:29.730515 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.730438    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c"
Oct 31 02:17:29.773690 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.773657    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6eae8ee-0ae3-4beb-9777-e63e8853a9cb-kube-api-access-859qt\" (UniqueName: \"kubernetes.io/projected/f6eae8ee-0ae3-4beb-9777-e63e8853a9cb-kube-api-access-859qt\") pod \"f6eae8ee-0ae3-4beb-9777-e63e8853a9cb\" (UID: \"f6eae8ee-0ae3-4beb-9777-e63e8853a9cb\") "
Oct 31 02:17:29.773890 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.773794    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6eae8ee-0ae3-4beb-9777-e63e8853a9cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6eae8ee-0ae3-4beb-9777-e63e8853a9cb-plugins-config-volume\") pod \"f6eae8ee-0ae3-4beb-9777-e63e8853a9cb\" (UID: \"f6eae8ee-0ae3-4beb-9777-e63e8853a9cb\") "
Oct 31 02:17:29.774242 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.774212    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6eae8ee-0ae3-4beb-9777-e63e8853a9cb-plugins-config-volume" pod "f6eae8ee-0ae3-4beb-9777-e63e8853a9cb" (UID: "f6eae8ee-0ae3-4beb-9777-e63e8853a9cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:29.775949 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.775927    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6eae8ee-0ae3-4beb-9777-e63e8853a9cb-kube-api-access-859qt" pod "f6eae8ee-0ae3-4beb-9777-e63e8853a9cb" (UID: "f6eae8ee-0ae3-4beb-9777-e63e8853a9cb"). InnerVolumeSpecName "kube-api-access-859qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:29.875746 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.875657    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/822c5afc-7c59-422d-a24d-d1b682127173-kube-api-access-gcf95\" (UniqueName: \"kubernetes.io/projected/822c5afc-7c59-422d-a24d-d1b682127173-kube-api-access-gcf95\") pod \"822c5afc-7c59-422d-a24d-d1b682127173\" (UID: \"822c5afc-7c59-422d-a24d-d1b682127173\") "
Oct 31 02:17:29.875899 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.875762    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/822c5afc-7c59-422d-a24d-d1b682127173-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/822c5afc-7c59-422d-a24d-d1b682127173-plugins-config-volume\") pod \"822c5afc-7c59-422d-a24d-d1b682127173\" (UID: \"822c5afc-7c59-422d-a24d-d1b682127173\") "
Oct 31 02:17:29.876197 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.876174    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/822c5afc-7c59-422d-a24d-d1b682127173-plugins-config-volume" pod "822c5afc-7c59-422d-a24d-d1b682127173" (UID: "822c5afc-7c59-422d-a24d-d1b682127173"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:29.876372 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.876357    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/822c5afc-7c59-422d-a24d-d1b682127173-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:29.876419 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.876377    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6eae8ee-0ae3-4beb-9777-e63e8853a9cb-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:29.876419 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.876386    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-859qt\" (UniqueName: \"kubernetes.io/projected/f6eae8ee-0ae3-4beb-9777-e63e8853a9cb-kube-api-access-859qt\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:29.877868 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.877849    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822c5afc-7c59-422d-a24d-d1b682127173-kube-api-access-gcf95" pod "822c5afc-7c59-422d-a24d-d1b682127173" (UID: "822c5afc-7c59-422d-a24d-d1b682127173"). InnerVolumeSpecName "kube-api-access-gcf95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:29.974755 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.974717    5650 generic.go:356] "Generic (PLEG): container finished" podID="f6eae8ee-0ae3-4beb-9777-e63e8853a9cb" containerID="c56ab6085f2b62b923413d5d1d4c09ead0d01ea13ce894601c817df50cf82034" exitCode=0
Oct 31 02:17:29.975013 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.974771    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd"
Oct 31 02:17:29.975013 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.974819    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd" event={"ID":"f6eae8ee-0ae3-4beb-9777-e63e8853a9cb","Type":"ContainerDied","Data":"c56ab6085f2b62b923413d5d1d4c09ead0d01ea13ce894601c817df50cf82034"}
Oct 31 02:17:29.975013 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.974900    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd" event={"ID":"f6eae8ee-0ae3-4beb-9777-e63e8853a9cb","Type":"ContainerDied","Data":"bf8b8a595598a05f222b42632b2fc4d42a7deb0099871241eafe5fa3d055b709"}
Oct 31 02:17:29.975013 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.974933    5650 scope.go:122] "RemoveContainer" containerID="c56ab6085f2b62b923413d5d1d4c09ead0d01ea13ce894601c817df50cf82034"
Oct 31 02:17:29.978174 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.978139    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcf95\" (UniqueName: \"kubernetes.io/projected/822c5afc-7c59-422d-a24d-d1b682127173-kube-api-access-gcf95\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:29.978749 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.978722    5650 generic.go:356] "Generic (PLEG): container finished" podID="822c5afc-7c59-422d-a24d-d1b682127173" containerID="e78e05002c6d23ce58d1cfb9c3e9757eb37eb4d09fa06648ad74b19993b31276" exitCode=0
Oct 31 02:17:29.978855 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.978793    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c" event={"ID":"822c5afc-7c59-422d-a24d-d1b682127173","Type":"ContainerDied","Data":"e78e05002c6d23ce58d1cfb9c3e9757eb37eb4d09fa06648ad74b19993b31276"}
Oct 31 02:17:29.978855 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.978839    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c" event={"ID":"822c5afc-7c59-422d-a24d-d1b682127173","Type":"ContainerDied","Data":"040058f7441110dfac0c2c0ab9d91346e4f592cdcdafff125ddca6192e0f7883"}
Oct 31 02:17:29.978949 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.978913    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c"
Oct 31 02:17:29.997252 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.997230    5650 scope.go:122] "RemoveContainer" containerID="c56ab6085f2b62b923413d5d1d4c09ead0d01ea13ce894601c817df50cf82034"
Oct 31 02:17:29.998792 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:17:29.998764    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c56ab6085f2b62b923413d5d1d4c09ead0d01ea13ce894601c817df50cf82034\": container with ID starting with c56ab6085f2b62b923413d5d1d4c09ead0d01ea13ce894601c817df50cf82034 not found: ID does not exist" containerID="c56ab6085f2b62b923413d5d1d4c09ead0d01ea13ce894601c817df50cf82034"
Oct 31 02:17:29.998915 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.998803    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c56ab6085f2b62b923413d5d1d4c09ead0d01ea13ce894601c817df50cf82034"} err="failed to get container status \"c56ab6085f2b62b923413d5d1d4c09ead0d01ea13ce894601c817df50cf82034\": rpc error: code = NotFound desc = could not find container \"c56ab6085f2b62b923413d5d1d4c09ead0d01ea13ce894601c817df50cf82034\": container with ID starting with c56ab6085f2b62b923413d5d1d4c09ead0d01ea13ce894601c817df50cf82034 not found: ID does not exist"
Oct 31 02:17:29.998915 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:29.998827    5650 scope.go:122] "RemoveContainer" containerID="e78e05002c6d23ce58d1cfb9c3e9757eb37eb4d09fa06648ad74b19993b31276"
Oct 31 02:17:30.009849 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:30.009822    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd"]
Oct 31 02:17:30.011814 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:30.011795    5650 scope.go:122] "RemoveContainer" containerID="e78e05002c6d23ce58d1cfb9c3e9757eb37eb4d09fa06648ad74b19993b31276"
Oct 31 02:17:30.012138 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:17:30.012113    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e78e05002c6d23ce58d1cfb9c3e9757eb37eb4d09fa06648ad74b19993b31276\": container with ID starting with e78e05002c6d23ce58d1cfb9c3e9757eb37eb4d09fa06648ad74b19993b31276 not found: ID does not exist" containerID="e78e05002c6d23ce58d1cfb9c3e9757eb37eb4d09fa06648ad74b19993b31276"
Oct 31 02:17:30.012211 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:30.012146    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e78e05002c6d23ce58d1cfb9c3e9757eb37eb4d09fa06648ad74b19993b31276"} err="failed to get container status \"e78e05002c6d23ce58d1cfb9c3e9757eb37eb4d09fa06648ad74b19993b31276\": rpc error: code = NotFound desc = could not find container \"e78e05002c6d23ce58d1cfb9c3e9757eb37eb4d09fa06648ad74b19993b31276\": container with ID starting with e78e05002c6d23ce58d1cfb9c3e9757eb37eb4d09fa06648ad74b19993b31276 not found: ID does not exist"
Oct 31 02:17:30.014021 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:30.013999    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d0c6ff-796996887-2lpxd"]
Oct 31 02:17:30.025790 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:30.025762    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c"]
Oct 31 02:17:30.030892 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:30.030868    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d0c6ff-b76b9fbb7-xlc4c"]
Oct 31 02:17:30.119813 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:30.119783    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="822c5afc-7c59-422d-a24d-d1b682127173" path="/var/lib/kubelet/pods/822c5afc-7c59-422d-a24d-d1b682127173/volumes"
Oct 31 02:17:30.120236 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:30.120223    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6eae8ee-0ae3-4beb-9777-e63e8853a9cb" path="/var/lib/kubelet/pods/f6eae8ee-0ae3-4beb-9777-e63e8853a9cb/volumes"
Oct 31 02:17:30.415208 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:30.415171    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g" podUID="44fe8812-7863-4904-bb34-5ee90f27db75" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:17:30.617847 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:30.617764    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"]
Oct 31 02:17:30.618072 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:30.618045    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm" podUID="d7cacf9b-ad38-4f4b-a68c-f331dc576304" containerName="istio-proxy" containerID="cri-o://290f7a6869447094d79a15dce4826d7edbcad39b3e0d358f4c7824132dde4380" gracePeriod=30
Oct 31 02:17:30.623393 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:30.623367    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"]
Oct 31 02:17:30.623713 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:30.623643    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9" podUID="e22feaff-a34e-4be5-b480-30b9db5c5d58" containerName="istio-proxy" containerID="cri-o://6b87ff1d036353fad9c9c4b880186d559c0f781aa3d0bb4867391d7514b5c801" gracePeriod=30
Oct 31 02:17:30.628030 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:30.628001    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"]
Oct 31 02:17:30.628246 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:30.628221    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc" podUID="c060bf51-21db-4f5b-889d-e655338a9bfa" containerName="istio-proxy" containerID="cri-o://522d6b8bb091e65c70faf765b617358495bbca78814bded32b97b96c2c336271" gracePeriod=30
Oct 31 02:17:31.328752 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:31.328721    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct"]
Oct 31 02:17:31.344231 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:31.344194    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw"]
Oct 31 02:17:32.101907 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:32.101874    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh"]
Oct 31 02:17:32.119268 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:32.119237    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5555cb-zpkxh"]
Oct 31 02:17:34.121592 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:34.121557    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68500858-f749-44a3-95d1-ebb88191c582" path="/var/lib/kubelet/pods/68500858-f749-44a3-95d1-ebb88191c582/volumes"
Oct 31 02:17:35.928161 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:35.928091    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:17:35.947188 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:35.947161    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:17:35.953018 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:35.952989    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:17:36.048511 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.048430    5650 generic.go:356] "Generic (PLEG): container finished" podID="e22feaff-a34e-4be5-b480-30b9db5c5d58" containerID="6b87ff1d036353fad9c9c4b880186d559c0f781aa3d0bb4867391d7514b5c801" exitCode=0
Oct 31 02:17:36.048511 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.048504    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9" event={"ID":"e22feaff-a34e-4be5-b480-30b9db5c5d58","Type":"ContainerDied","Data":"6b87ff1d036353fad9c9c4b880186d559c0f781aa3d0bb4867391d7514b5c801"}
Oct 31 02:17:36.048743 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.048556    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9" event={"ID":"e22feaff-a34e-4be5-b480-30b9db5c5d58","Type":"ContainerDied","Data":"5408a63fb55d0b71db19d8d31ed1f35c46a9ab935031382f858c3f0ed2cea202"}
Oct 31 02:17:36.048743 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.048580    5650 scope.go:122] "RemoveContainer" containerID="6b87ff1d036353fad9c9c4b880186d559c0f781aa3d0bb4867391d7514b5c801"
Oct 31 02:17:36.048743 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.048512    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"
Oct 31 02:17:36.050416 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.050394    5650 generic.go:356] "Generic (PLEG): container finished" podID="d7cacf9b-ad38-4f4b-a68c-f331dc576304" containerID="290f7a6869447094d79a15dce4826d7edbcad39b3e0d358f4c7824132dde4380" exitCode=0
Oct 31 02:17:36.050549 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.050487    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"
Oct 31 02:17:36.050631 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.050488    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm" event={"ID":"d7cacf9b-ad38-4f4b-a68c-f331dc576304","Type":"ContainerDied","Data":"290f7a6869447094d79a15dce4826d7edbcad39b3e0d358f4c7824132dde4380"}
Oct 31 02:17:36.050711 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.050697    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm" event={"ID":"d7cacf9b-ad38-4f4b-a68c-f331dc576304","Type":"ContainerDied","Data":"919eb51cb3af56b5bd4a7e0c175c9bbc6ae4ae525a5a763ae94c6f39211214c9"}
Oct 31 02:17:36.052581 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.052559    5650 generic.go:356] "Generic (PLEG): container finished" podID="c060bf51-21db-4f5b-889d-e655338a9bfa" containerID="522d6b8bb091e65c70faf765b617358495bbca78814bded32b97b96c2c336271" exitCode=0
Oct 31 02:17:36.052717 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.052605    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc" event={"ID":"c060bf51-21db-4f5b-889d-e655338a9bfa","Type":"ContainerDied","Data":"522d6b8bb091e65c70faf765b617358495bbca78814bded32b97b96c2c336271"}
Oct 31 02:17:36.052717 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.052620    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"
Oct 31 02:17:36.052717 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.052641    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc" event={"ID":"c060bf51-21db-4f5b-889d-e655338a9bfa","Type":"ContainerDied","Data":"a48695e28a959d67626f196ff8187a55f9ecf96f52fcb2e295cfcff12d7a17bb"}
Oct 31 02:17:36.060284 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.060267    5650 scope.go:122] "RemoveContainer" containerID="6b87ff1d036353fad9c9c4b880186d559c0f781aa3d0bb4867391d7514b5c801"
Oct 31 02:17:36.060585 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:17:36.060566    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b87ff1d036353fad9c9c4b880186d559c0f781aa3d0bb4867391d7514b5c801\": container with ID starting with 6b87ff1d036353fad9c9c4b880186d559c0f781aa3d0bb4867391d7514b5c801 not found: ID does not exist" containerID="6b87ff1d036353fad9c9c4b880186d559c0f781aa3d0bb4867391d7514b5c801"
Oct 31 02:17:36.060664 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.060598    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b87ff1d036353fad9c9c4b880186d559c0f781aa3d0bb4867391d7514b5c801"} err="failed to get container status \"6b87ff1d036353fad9c9c4b880186d559c0f781aa3d0bb4867391d7514b5c801\": rpc error: code = NotFound desc = could not find container \"6b87ff1d036353fad9c9c4b880186d559c0f781aa3d0bb4867391d7514b5c801\": container with ID starting with 6b87ff1d036353fad9c9c4b880186d559c0f781aa3d0bb4867391d7514b5c801 not found: ID does not exist"
Oct 31 02:17:36.060664 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.060622    5650 scope.go:122] "RemoveContainer" containerID="290f7a6869447094d79a15dce4826d7edbcad39b3e0d358f4c7824132dde4380"
Oct 31 02:17:36.069241 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.069215    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-token\" (UniqueName: \"kubernetes.io/projected/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-token\") pod \"c060bf51-21db-4f5b-889d-e655338a9bfa\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") "
Oct 31 02:17:36.069359 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.069264    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7cacf9b-ad38-4f4b-a68c-f331dc576304-kube-api-access-rzljs\" (UniqueName: \"kubernetes.io/projected/d7cacf9b-ad38-4f4b-a68c-f331dc576304-kube-api-access-rzljs\") pod \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") "
Oct 31 02:17:36.069359 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.069298    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-workload-certs\") pod \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") "
Oct 31 02:17:36.069359 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.069329    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e22feaff-a34e-4be5-b480-30b9db5c5d58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e22feaff-a34e-4be5-b480-30b9db5c5d58-istiod-ca-cert\") pod \"e22feaff-a34e-4be5-b480-30b9db5c5d58\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") "
Oct 31 02:17:36.069525 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.069370    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-credential-socket\") pod \"e22feaff-a34e-4be5-b480-30b9db5c5d58\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") "
Oct 31 02:17:36.069525 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.069397    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-envoy\") pod \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") "
Oct 31 02:17:36.069525 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.069511    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-data\") pod \"c060bf51-21db-4f5b-889d-e655338a9bfa\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") "
Oct 31 02:17:36.069855 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.069542    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e22feaff-a34e-4be5-b480-30b9db5c5d58-kube-api-access-7tbkx\" (UniqueName: \"kubernetes.io/projected/e22feaff-a34e-4be5-b480-30b9db5c5d58-kube-api-access-7tbkx\") pod \"e22feaff-a34e-4be5-b480-30b9db5c5d58\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") "
Oct 31 02:17:36.069855 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.069603    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-podinfo\") pod \"e22feaff-a34e-4be5-b480-30b9db5c5d58\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") "
Oct 31 02:17:36.069855 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.069635    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-workload-socket\") pod \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") "
Oct 31 02:17:36.069855 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.069677    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-credential-socket\") pod \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") "
Oct 31 02:17:36.069855 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.069706    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-workload-certs" pod "d7cacf9b-ad38-4f4b-a68c-f331dc576304" (UID: "d7cacf9b-ad38-4f4b-a68c-f331dc576304"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:36.069855 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.069721    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-token\" (UniqueName: \"kubernetes.io/projected/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-token\") pod \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") "
Oct 31 02:17:36.069855 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.069762    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-podinfo\") pod \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") "
Oct 31 02:17:36.069855 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.069844    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-data\") pod \"e22feaff-a34e-4be5-b480-30b9db5c5d58\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") "
Oct 31 02:17:36.070699 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.069871    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-credential-socket" pod "e22feaff-a34e-4be5-b480-30b9db5c5d58" (UID: "e22feaff-a34e-4be5-b480-30b9db5c5d58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:36.070699 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.069900    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-envoy\") pod \"c060bf51-21db-4f5b-889d-e655338a9bfa\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") "
Oct 31 02:17:36.070699 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.069949    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c060bf51-21db-4f5b-889d-e655338a9bfa-kube-api-access-42b42\" (UniqueName: \"kubernetes.io/projected/c060bf51-21db-4f5b-889d-e655338a9bfa-kube-api-access-42b42\") pod \"c060bf51-21db-4f5b-889d-e655338a9bfa\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") "
Oct 31 02:17:36.070699 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.069988    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-data\") pod \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") "
Oct 31 02:17:36.070699 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.070025    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-token\" (UniqueName: \"kubernetes.io/projected/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-token\") pod \"e22feaff-a34e-4be5-b480-30b9db5c5d58\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") "
Oct 31 02:17:36.070699 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.070067    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-workload-socket\") pod \"e22feaff-a34e-4be5-b480-30b9db5c5d58\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") "
Oct 31 02:17:36.070699 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.070124    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c060bf51-21db-4f5b-889d-e655338a9bfa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c060bf51-21db-4f5b-889d-e655338a9bfa-istiod-ca-cert\") pod \"c060bf51-21db-4f5b-889d-e655338a9bfa\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") "
Oct 31 02:17:36.070699 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.070188    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e22feaff-a34e-4be5-b480-30b9db5c5d58-istiod-ca-cert" pod "e22feaff-a34e-4be5-b480-30b9db5c5d58" (UID: "e22feaff-a34e-4be5-b480-30b9db5c5d58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:36.070699 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.070196    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istiod-ca-cert\") pod \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\" (UID: \"d7cacf9b-ad38-4f4b-a68c-f331dc576304\") "
Oct 31 02:17:36.070699 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.070382    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-credential-socket\") pod \"c060bf51-21db-4f5b-889d-e655338a9bfa\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") "
Oct 31 02:17:36.070699 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.070414    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-workload-certs\") pod \"c060bf51-21db-4f5b-889d-e655338a9bfa\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") "
Oct 31 02:17:36.070699 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.070477    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-workload-certs\") pod \"e22feaff-a34e-4be5-b480-30b9db5c5d58\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") "
Oct 31 02:17:36.070699 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.070513    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-podinfo\") pod \"c060bf51-21db-4f5b-889d-e655338a9bfa\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") "
Oct 31 02:17:36.070699 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.070558    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-workload-socket\") pod \"c060bf51-21db-4f5b-889d-e655338a9bfa\" (UID: \"c060bf51-21db-4f5b-889d-e655338a9bfa\") "
Oct 31 02:17:36.070699 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.070610    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-envoy\") pod \"e22feaff-a34e-4be5-b480-30b9db5c5d58\" (UID: \"e22feaff-a34e-4be5-b480-30b9db5c5d58\") "
Oct 31 02:17:36.071792 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.071303    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-workload-socket" pod "d7cacf9b-ad38-4f4b-a68c-f331dc576304" (UID: "d7cacf9b-ad38-4f4b-a68c-f331dc576304"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:36.071792 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.071335    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-data" pod "c060bf51-21db-4f5b-889d-e655338a9bfa" (UID: "c060bf51-21db-4f5b-889d-e655338a9bfa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:36.071792 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.071475    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-data" pod "e22feaff-a34e-4be5-b480-30b9db5c5d58" (UID: "e22feaff-a34e-4be5-b480-30b9db5c5d58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:36.071792 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.071642    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-workload-certs" pod "e22feaff-a34e-4be5-b480-30b9db5c5d58" (UID: "e22feaff-a34e-4be5-b480-30b9db5c5d58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:36.072018 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.071823    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-workload-socket" pod "c060bf51-21db-4f5b-889d-e655338a9bfa" (UID: "c060bf51-21db-4f5b-889d-e655338a9bfa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:36.072018 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.071840    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-credential-socket" pod "c060bf51-21db-4f5b-889d-e655338a9bfa" (UID: "c060bf51-21db-4f5b-889d-e655338a9bfa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:36.072018 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.071958    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-workload-socket" pod "e22feaff-a34e-4be5-b480-30b9db5c5d58" (UID: "e22feaff-a34e-4be5-b480-30b9db5c5d58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:36.072176 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.072107    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c060bf51-21db-4f5b-889d-e655338a9bfa-istiod-ca-cert" pod "c060bf51-21db-4f5b-889d-e655338a9bfa" (UID: "c060bf51-21db-4f5b-889d-e655338a9bfa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:36.072763 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.072735    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.072888 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.072769    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e22feaff-a34e-4be5-b480-30b9db5c5d58-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.072888 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.072786    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.072888 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.072800    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.072888 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.072815    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.072888 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.072829    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.072888 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.072844    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.072888 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.072858    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c060bf51-21db-4f5b-889d-e655338a9bfa-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.072888 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.072872    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.072888 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.072887    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.073586 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.072902    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.073586 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.073095    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-data" pod "d7cacf9b-ad38-4f4b-a68c-f331dc576304" (UID: "d7cacf9b-ad38-4f4b-a68c-f331dc576304"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:36.073586 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.073152    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-workload-certs" pod "c060bf51-21db-4f5b-889d-e655338a9bfa" (UID: "c060bf51-21db-4f5b-889d-e655338a9bfa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:36.073586 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.073483    5650 scope.go:122] "RemoveContainer" containerID="290f7a6869447094d79a15dce4826d7edbcad39b3e0d358f4c7824132dde4380"
Oct 31 02:17:36.073890 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.073751    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-credential-socket" pod "d7cacf9b-ad38-4f4b-a68c-f331dc576304" (UID: "d7cacf9b-ad38-4f4b-a68c-f331dc576304"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:36.073890 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:17:36.073871    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"290f7a6869447094d79a15dce4826d7edbcad39b3e0d358f4c7824132dde4380\": container with ID starting with 290f7a6869447094d79a15dce4826d7edbcad39b3e0d358f4c7824132dde4380 not found: ID does not exist" containerID="290f7a6869447094d79a15dce4826d7edbcad39b3e0d358f4c7824132dde4380"
Oct 31 02:17:36.074025 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.073905    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"290f7a6869447094d79a15dce4826d7edbcad39b3e0d358f4c7824132dde4380"} err="failed to get container status \"290f7a6869447094d79a15dce4826d7edbcad39b3e0d358f4c7824132dde4380\": rpc error: code = NotFound desc = could not find container \"290f7a6869447094d79a15dce4826d7edbcad39b3e0d358f4c7824132dde4380\": container with ID starting with 290f7a6869447094d79a15dce4826d7edbcad39b3e0d358f4c7824132dde4380 not found: ID does not exist"
Oct 31 02:17:36.074025 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.073932    5650 scope.go:122] "RemoveContainer" containerID="522d6b8bb091e65c70faf765b617358495bbca78814bded32b97b96c2c336271"
Oct 31 02:17:36.074176 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.074151    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-podinfo" pod "e22feaff-a34e-4be5-b480-30b9db5c5d58" (UID: "e22feaff-a34e-4be5-b480-30b9db5c5d58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:17:36.074252 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.074193    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istiod-ca-cert" pod "d7cacf9b-ad38-4f4b-a68c-f331dc576304" (UID: "d7cacf9b-ad38-4f4b-a68c-f331dc576304"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:36.074509 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.074433    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-token" pod "e22feaff-a34e-4be5-b480-30b9db5c5d58" (UID: "e22feaff-a34e-4be5-b480-30b9db5c5d58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:36.074614 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.074557    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-token" pod "c060bf51-21db-4f5b-889d-e655338a9bfa" (UID: "c060bf51-21db-4f5b-889d-e655338a9bfa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:36.074894 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.074859    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-envoy" pod "e22feaff-a34e-4be5-b480-30b9db5c5d58" (UID: "e22feaff-a34e-4be5-b480-30b9db5c5d58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:36.075140 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.075100    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-envoy" pod "c060bf51-21db-4f5b-889d-e655338a9bfa" (UID: "c060bf51-21db-4f5b-889d-e655338a9bfa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:36.075140 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.075118    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-envoy" pod "d7cacf9b-ad38-4f4b-a68c-f331dc576304" (UID: "d7cacf9b-ad38-4f4b-a68c-f331dc576304"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:36.075285 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.075176    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e22feaff-a34e-4be5-b480-30b9db5c5d58-kube-api-access-7tbkx" pod "e22feaff-a34e-4be5-b480-30b9db5c5d58" (UID: "e22feaff-a34e-4be5-b480-30b9db5c5d58"). InnerVolumeSpecName "kube-api-access-7tbkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:36.075285 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.075231    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-token" pod "d7cacf9b-ad38-4f4b-a68c-f331dc576304" (UID: "d7cacf9b-ad38-4f4b-a68c-f331dc576304"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:36.075713 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.075642    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-podinfo" pod "c060bf51-21db-4f5b-889d-e655338a9bfa" (UID: "c060bf51-21db-4f5b-889d-e655338a9bfa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:17:36.076174 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.076153    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c060bf51-21db-4f5b-889d-e655338a9bfa-kube-api-access-42b42" pod "c060bf51-21db-4f5b-889d-e655338a9bfa" (UID: "c060bf51-21db-4f5b-889d-e655338a9bfa"). InnerVolumeSpecName "kube-api-access-42b42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:36.076384 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.076358    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7cacf9b-ad38-4f4b-a68c-f331dc576304-kube-api-access-rzljs" pod "d7cacf9b-ad38-4f4b-a68c-f331dc576304" (UID: "d7cacf9b-ad38-4f4b-a68c-f331dc576304"). InnerVolumeSpecName "kube-api-access-rzljs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:36.076759 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.076739    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-podinfo" pod "d7cacf9b-ad38-4f4b-a68c-f331dc576304" (UID: "d7cacf9b-ad38-4f4b-a68c-f331dc576304"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:17:36.085265 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.085242    5650 scope.go:122] "RemoveContainer" containerID="749f9e14419765c55ba413d5e5d93be4f909775c40d3ecb3b30a3d533f6b4c57"
Oct 31 02:17:36.095490 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.095466    5650 scope.go:122] "RemoveContainer" containerID="522d6b8bb091e65c70faf765b617358495bbca78814bded32b97b96c2c336271"
Oct 31 02:17:36.095790 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:17:36.095771    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"522d6b8bb091e65c70faf765b617358495bbca78814bded32b97b96c2c336271\": container with ID starting with 522d6b8bb091e65c70faf765b617358495bbca78814bded32b97b96c2c336271 not found: ID does not exist" containerID="522d6b8bb091e65c70faf765b617358495bbca78814bded32b97b96c2c336271"
Oct 31 02:17:36.095849 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.095798    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"522d6b8bb091e65c70faf765b617358495bbca78814bded32b97b96c2c336271"} err="failed to get container status \"522d6b8bb091e65c70faf765b617358495bbca78814bded32b97b96c2c336271\": rpc error: code = NotFound desc = could not find container \"522d6b8bb091e65c70faf765b617358495bbca78814bded32b97b96c2c336271\": container with ID starting with 522d6b8bb091e65c70faf765b617358495bbca78814bded32b97b96c2c336271 not found: ID does not exist"
Oct 31 02:17:36.095849 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.095815    5650 scope.go:122] "RemoveContainer" containerID="749f9e14419765c55ba413d5e5d93be4f909775c40d3ecb3b30a3d533f6b4c57"
Oct 31 02:17:36.096119 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:17:36.096099    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"749f9e14419765c55ba413d5e5d93be4f909775c40d3ecb3b30a3d533f6b4c57\": container with ID starting with 749f9e14419765c55ba413d5e5d93be4f909775c40d3ecb3b30a3d533f6b4c57 not found: ID does not exist" containerID="749f9e14419765c55ba413d5e5d93be4f909775c40d3ecb3b30a3d533f6b4c57"
Oct 31 02:17:36.096172 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.096127    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749f9e14419765c55ba413d5e5d93be4f909775c40d3ecb3b30a3d533f6b4c57"} err="failed to get container status \"749f9e14419765c55ba413d5e5d93be4f909775c40d3ecb3b30a3d533f6b4c57\": rpc error: code = NotFound desc = could not find container \"749f9e14419765c55ba413d5e5d93be4f909775c40d3ecb3b30a3d533f6b4c57\": container with ID starting with 749f9e14419765c55ba413d5e5d93be4f909775c40d3ecb3b30a3d533f6b4c57 not found: ID does not exist"
Oct 31 02:17:36.174197 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.174160    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.174197 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.174196    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.174197 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.174206    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.174472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.174215    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.174472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.174225    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.174472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.174234    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzljs\" (UniqueName: \"kubernetes.io/projected/d7cacf9b-ad38-4f4b-a68c-f331dc576304-kube-api-access-rzljs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.174472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.174242    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.174472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.174251    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tbkx\" (UniqueName: \"kubernetes.io/projected/e22feaff-a34e-4be5-b480-30b9db5c5d58-kube-api-access-7tbkx\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.174472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.174259    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.174472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.174268    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.174472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.174277    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.174472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.174286    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.174472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.174295    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c060bf51-21db-4f5b-889d-e655338a9bfa-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.174472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.174304    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42b42\" (UniqueName: \"kubernetes.io/projected/c060bf51-21db-4f5b-889d-e655338a9bfa-kube-api-access-42b42\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.174472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.174316    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7cacf9b-ad38-4f4b-a68c-f331dc576304-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.174472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.174324    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e22feaff-a34e-4be5-b480-30b9db5c5d58-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.329693 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.329646    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct" podUID="218ba464-377e-48e6-a904-59aac3818f28" containerName="payload-processing" containerID="cri-o://863072f7e6633458c783512a207f7c89ec36a773dea14b3e288a763c06d44930" gracePeriod=25
Oct 31 02:17:36.344818 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.344780    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw" podUID="9fc48192-d1e7-4f3d-8285-3c0d98465241" containerName="payload-pre-processing" containerID="cri-o://e9e36b549647e6d265e3691453d27f7bf1e8f76db88665263719225772fb9e96" gracePeriod=25
Oct 31 02:17:36.438998 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.438301    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"]
Oct 31 02:17:36.443186 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.443161    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-25tr9"]
Oct 31 02:17:36.457837 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.457793    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"]
Oct 31 02:17:36.463677 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.463647    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-4bxbc"]
Oct 31 02:17:36.478895 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.478865    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"]
Oct 31 02:17:36.483297 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.483266    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gvtqm"]
Oct 31 02:17:36.610022 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.609987    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct"
Oct 31 02:17:36.706636 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.706609    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw"
Oct 31 02:17:36.780774 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.780736    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/218ba464-377e-48e6-a904-59aac3818f28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/218ba464-377e-48e6-a904-59aac3818f28-plugins-config-volume\") pod \"218ba464-377e-48e6-a904-59aac3818f28\" (UID: \"218ba464-377e-48e6-a904-59aac3818f28\") "
Oct 31 02:17:36.780954 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.780939    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/218ba464-377e-48e6-a904-59aac3818f28-kube-api-access-5hlxt\" (UniqueName: \"kubernetes.io/projected/218ba464-377e-48e6-a904-59aac3818f28-kube-api-access-5hlxt\") pod \"218ba464-377e-48e6-a904-59aac3818f28\" (UID: \"218ba464-377e-48e6-a904-59aac3818f28\") "
Oct 31 02:17:36.781191 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.781157    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/218ba464-377e-48e6-a904-59aac3818f28-plugins-config-volume" pod "218ba464-377e-48e6-a904-59aac3818f28" (UID: "218ba464-377e-48e6-a904-59aac3818f28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:36.781556 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.781527    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/218ba464-377e-48e6-a904-59aac3818f28-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.783025 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.783001    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/218ba464-377e-48e6-a904-59aac3818f28-kube-api-access-5hlxt" pod "218ba464-377e-48e6-a904-59aac3818f28" (UID: "218ba464-377e-48e6-a904-59aac3818f28"). InnerVolumeSpecName "kube-api-access-5hlxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:36.882201 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.882096    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc48192-d1e7-4f3d-8285-3c0d98465241-kube-api-access-n6lf6\" (UniqueName: \"kubernetes.io/projected/9fc48192-d1e7-4f3d-8285-3c0d98465241-kube-api-access-n6lf6\") pod \"9fc48192-d1e7-4f3d-8285-3c0d98465241\" (UID: \"9fc48192-d1e7-4f3d-8285-3c0d98465241\") "
Oct 31 02:17:36.882390 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.882211    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fc48192-d1e7-4f3d-8285-3c0d98465241-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc48192-d1e7-4f3d-8285-3c0d98465241-plugins-config-volume\") pod \"9fc48192-d1e7-4f3d-8285-3c0d98465241\" (UID: \"9fc48192-d1e7-4f3d-8285-3c0d98465241\") "
Oct 31 02:17:36.882626 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.882596    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc48192-d1e7-4f3d-8285-3c0d98465241-plugins-config-volume" pod "9fc48192-d1e7-4f3d-8285-3c0d98465241" (UID: "9fc48192-d1e7-4f3d-8285-3c0d98465241"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:36.882853 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.882837    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fc48192-d1e7-4f3d-8285-3c0d98465241-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.882907 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.882868    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hlxt\" (UniqueName: \"kubernetes.io/projected/218ba464-377e-48e6-a904-59aac3818f28-kube-api-access-5hlxt\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:36.884501 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.884473    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc48192-d1e7-4f3d-8285-3c0d98465241-kube-api-access-n6lf6" pod "9fc48192-d1e7-4f3d-8285-3c0d98465241" (UID: "9fc48192-d1e7-4f3d-8285-3c0d98465241"). InnerVolumeSpecName "kube-api-access-n6lf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:36.983779 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:36.983741    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6lf6\" (UniqueName: \"kubernetes.io/projected/9fc48192-d1e7-4f3d-8285-3c0d98465241-kube-api-access-n6lf6\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:37.017876 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:37.017836    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc"]
Oct 31 02:17:37.032097 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:37.032063    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg"]
Oct 31 02:17:37.070243 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:37.070210    5650 generic.go:356] "Generic (PLEG): container finished" podID="218ba464-377e-48e6-a904-59aac3818f28" containerID="863072f7e6633458c783512a207f7c89ec36a773dea14b3e288a763c06d44930" exitCode=0
Oct 31 02:17:37.070479 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:37.070376    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct" event={"ID":"218ba464-377e-48e6-a904-59aac3818f28","Type":"ContainerDied","Data":"863072f7e6633458c783512a207f7c89ec36a773dea14b3e288a763c06d44930"}
Oct 31 02:17:37.070479 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:37.070431    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct" event={"ID":"218ba464-377e-48e6-a904-59aac3818f28","Type":"ContainerDied","Data":"5327dab662093f9321241b3e80641e9be56f9d92ba473d0ff19bf7818026b17c"}
Oct 31 02:17:37.070685 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:37.070499    5650 scope.go:122] "RemoveContainer" containerID="863072f7e6633458c783512a207f7c89ec36a773dea14b3e288a763c06d44930"
Oct 31 02:17:37.070685 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:37.070668    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct"
Oct 31 02:17:37.098650 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:37.098600    5650 generic.go:356] "Generic (PLEG): container finished" podID="9fc48192-d1e7-4f3d-8285-3c0d98465241" containerID="e9e36b549647e6d265e3691453d27f7bf1e8f76db88665263719225772fb9e96" exitCode=0
Oct 31 02:17:37.098805 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:37.098690    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw" event={"ID":"9fc48192-d1e7-4f3d-8285-3c0d98465241","Type":"ContainerDied","Data":"e9e36b549647e6d265e3691453d27f7bf1e8f76db88665263719225772fb9e96"}
Oct 31 02:17:37.098805 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:37.098740    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw" event={"ID":"9fc48192-d1e7-4f3d-8285-3c0d98465241","Type":"ContainerDied","Data":"31af3d7e075a92e009494d80afc3d51280d11db925af2b76081abec5bfaf4f21"}
Oct 31 02:17:37.098928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:37.098831    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw"
Oct 31 02:17:37.107904 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:37.107843    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct"]
Oct 31 02:17:37.109248 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:37.109226    5650 scope.go:122] "RemoveContainer" containerID="863072f7e6633458c783512a207f7c89ec36a773dea14b3e288a763c06d44930"
Oct 31 02:17:37.109621 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:17:37.109593    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"863072f7e6633458c783512a207f7c89ec36a773dea14b3e288a763c06d44930\": container with ID starting with 863072f7e6633458c783512a207f7c89ec36a773dea14b3e288a763c06d44930 not found: ID does not exist" containerID="863072f7e6633458c783512a207f7c89ec36a773dea14b3e288a763c06d44930"
Oct 31 02:17:37.109692 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:37.109635    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"863072f7e6633458c783512a207f7c89ec36a773dea14b3e288a763c06d44930"} err="failed to get container status \"863072f7e6633458c783512a207f7c89ec36a773dea14b3e288a763c06d44930\": rpc error: code = NotFound desc = could not find container \"863072f7e6633458c783512a207f7c89ec36a773dea14b3e288a763c06d44930\": container with ID starting with 863072f7e6633458c783512a207f7c89ec36a773dea14b3e288a763c06d44930 not found: ID does not exist"
Oct 31 02:17:37.109692 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:37.109662    5650 scope.go:122] "RemoveContainer" containerID="e9e36b549647e6d265e3691453d27f7bf1e8f76db88665263719225772fb9e96"
Oct 31 02:17:37.114920 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:37.114898    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5555cb-7f64c54897-vc7ct"]
Oct 31 02:17:37.124511 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:37.124475    5650 scope.go:122] "RemoveContainer" containerID="e9e36b549647e6d265e3691453d27f7bf1e8f76db88665263719225772fb9e96"
Oct 31 02:17:37.124846 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:17:37.124825    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9e36b549647e6d265e3691453d27f7bf1e8f76db88665263719225772fb9e96\": container with ID starting with e9e36b549647e6d265e3691453d27f7bf1e8f76db88665263719225772fb9e96 not found: ID does not exist" containerID="e9e36b549647e6d265e3691453d27f7bf1e8f76db88665263719225772fb9e96"
Oct 31 02:17:37.124923 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:37.124856    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e36b549647e6d265e3691453d27f7bf1e8f76db88665263719225772fb9e96"} err="failed to get container status \"e9e36b549647e6d265e3691453d27f7bf1e8f76db88665263719225772fb9e96\": rpc error: code = NotFound desc = could not find container \"e9e36b549647e6d265e3691453d27f7bf1e8f76db88665263719225772fb9e96\": container with ID starting with e9e36b549647e6d265e3691453d27f7bf1e8f76db88665263719225772fb9e96 not found: ID does not exist"
Oct 31 02:17:37.132600 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:37.132376    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw"]
Oct 31 02:17:37.136329 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:37.136304    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5555cb-5495656679-p4fzw"]
Oct 31 02:17:38.139203 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:38.139161    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="218ba464-377e-48e6-a904-59aac3818f28" path="/var/lib/kubelet/pods/218ba464-377e-48e6-a904-59aac3818f28/volumes"
Oct 31 02:17:38.139817 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:38.139794    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fc48192-d1e7-4f3d-8285-3c0d98465241" path="/var/lib/kubelet/pods/9fc48192-d1e7-4f3d-8285-3c0d98465241/volumes"
Oct 31 02:17:38.140289 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:38.140269    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c060bf51-21db-4f5b-889d-e655338a9bfa" path="/var/lib/kubelet/pods/c060bf51-21db-4f5b-889d-e655338a9bfa/volumes"
Oct 31 02:17:38.141125 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:38.141104    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7cacf9b-ad38-4f4b-a68c-f331dc576304" path="/var/lib/kubelet/pods/d7cacf9b-ad38-4f4b-a68c-f331dc576304/volumes"
Oct 31 02:17:38.141624 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:38.141607    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e22feaff-a34e-4be5-b480-30b9db5c5d58" path="/var/lib/kubelet/pods/e22feaff-a34e-4be5-b480-30b9db5c5d58/volumes"
Oct 31 02:17:40.242560 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:40.242523    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"]
Oct 31 02:17:40.243089 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:40.242744    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m" podUID="17e6d59f-a66f-4c5c-aff8-8725deabb93f" containerName="istio-proxy" containerID="cri-o://d9882cb0bd8d18f1b22605da5a609e49f80bfdf426cbb280e8f9c1369234a920" gracePeriod=30
Oct 31 02:17:40.415326 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:40.415285    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g" podUID="44fe8812-7863-4904-bb34-5ee90f27db75" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:17:41.585715 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:41.585675    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh"]
Oct 31 02:17:41.604116 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:41.604074    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk"]
Oct 31 02:17:42.018235 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:42.018198    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc" podUID="6a1974b0-604c-4805-b965-0b0cfcf89b30" containerName="payload-processing" containerID="cri-o://2e06fbdd7d5a1688ac2c939b0f5d8c81211fa1f6d10390af57aa348b3c95d1a3" gracePeriod=25
Oct 31 02:17:42.032547 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:42.032505    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg" podUID="c24d5105-6144-4a56-8aad-39be92445ea5" containerName="payload-pre-processing" containerID="cri-o://f294c53ed00eb8a48dac3d12ab0b8e3a277f398c670d457cf9e423a9fc7a9d6e" gracePeriod=25
Oct 31 02:17:42.726386 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:42.726348    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg" podUID="c24d5105-6144-4a56-8aad-39be92445ea5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:17:42.726826 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:42.726376    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc" podUID="6a1974b0-604c-4805-b965-0b0cfcf89b30" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:17:44.215954 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:44.215923    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g"
Oct 31 02:17:44.232571 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:44.232541    5650 generic.go:356] "Generic (PLEG): container finished" podID="44fe8812-7863-4904-bb34-5ee90f27db75" containerID="49bbaf641308eeb3b1a50644103c0959e57337e28375476c0b612858d26e3bd5" exitCode=0
Oct 31 02:17:44.232744 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:44.232583    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g"
Oct 31 02:17:44.232744 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:44.232618    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g" event={"ID":"44fe8812-7863-4904-bb34-5ee90f27db75","Type":"ContainerDied","Data":"49bbaf641308eeb3b1a50644103c0959e57337e28375476c0b612858d26e3bd5"}
Oct 31 02:17:44.232744 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:44.232676    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g" event={"ID":"44fe8812-7863-4904-bb34-5ee90f27db75","Type":"ContainerDied","Data":"97d93a372b11cfc3839c70f1a597ab809171e6eb9984b9addbec7f367382bd89"}
Oct 31 02:17:44.232744 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:44.232687    5650 scope.go:122] "RemoveContainer" containerID="49bbaf641308eeb3b1a50644103c0959e57337e28375476c0b612858d26e3bd5"
Oct 31 02:17:44.248912 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:44.248308    5650 scope.go:122] "RemoveContainer" containerID="49bbaf641308eeb3b1a50644103c0959e57337e28375476c0b612858d26e3bd5"
Oct 31 02:17:44.248912 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:17:44.248837    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49bbaf641308eeb3b1a50644103c0959e57337e28375476c0b612858d26e3bd5\": container with ID starting with 49bbaf641308eeb3b1a50644103c0959e57337e28375476c0b612858d26e3bd5 not found: ID does not exist" containerID="49bbaf641308eeb3b1a50644103c0959e57337e28375476c0b612858d26e3bd5"
Oct 31 02:17:44.248912 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:44.248886    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49bbaf641308eeb3b1a50644103c0959e57337e28375476c0b612858d26e3bd5"} err="failed to get container status \"49bbaf641308eeb3b1a50644103c0959e57337e28375476c0b612858d26e3bd5\": rpc error: code = NotFound desc = could not find container \"49bbaf641308eeb3b1a50644103c0959e57337e28375476c0b612858d26e3bd5\": container with ID starting with 49bbaf641308eeb3b1a50644103c0959e57337e28375476c0b612858d26e3bd5 not found: ID does not exist"
Oct 31 02:17:44.321992 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:44.321950    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44fe8812-7863-4904-bb34-5ee90f27db75-kube-api-access-28ml8\" (UniqueName: \"kubernetes.io/projected/44fe8812-7863-4904-bb34-5ee90f27db75-kube-api-access-28ml8\") pod \"44fe8812-7863-4904-bb34-5ee90f27db75\" (UID: \"44fe8812-7863-4904-bb34-5ee90f27db75\") "
Oct 31 02:17:44.322247 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:44.322234    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44fe8812-7863-4904-bb34-5ee90f27db75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44fe8812-7863-4904-bb34-5ee90f27db75-plugins-config-volume\") pod \"44fe8812-7863-4904-bb34-5ee90f27db75\" (UID: \"44fe8812-7863-4904-bb34-5ee90f27db75\") "
Oct 31 02:17:44.322612 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:44.322590    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44fe8812-7863-4904-bb34-5ee90f27db75-plugins-config-volume" pod "44fe8812-7863-4904-bb34-5ee90f27db75" (UID: "44fe8812-7863-4904-bb34-5ee90f27db75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:44.324392 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:44.324366    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44fe8812-7863-4904-bb34-5ee90f27db75-kube-api-access-28ml8" pod "44fe8812-7863-4904-bb34-5ee90f27db75" (UID: "44fe8812-7863-4904-bb34-5ee90f27db75"). InnerVolumeSpecName "kube-api-access-28ml8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:44.426337 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:44.426287    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44fe8812-7863-4904-bb34-5ee90f27db75-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:44.426337 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:44.426338    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28ml8\" (UniqueName: \"kubernetes.io/projected/44fe8812-7863-4904-bb34-5ee90f27db75-kube-api-access-28ml8\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:44.558775 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:44.558743    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g"]
Oct 31 02:17:44.563605 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:44.563563    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-91b6b1-6865976887-dfq2g"]
Oct 31 02:17:45.373660 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.373628    5650 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 31 02:17:45.374078 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.373685    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m" podUID="17e6d59f-a66f-4c5c-aff8-8725deabb93f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 31 02:17:45.548662 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.548631    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 02:17:45.643877 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.643837    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-podinfo\") pod \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") "
Oct 31 02:17:45.644088 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.643931    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-token\" (UniqueName: \"kubernetes.io/projected/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-token\") pod \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") "
Oct 31 02:17:45.644088 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.644019    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-workload-certs\") pod \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") "
Oct 31 02:17:45.644088 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.644049    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-envoy\") pod \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") "
Oct 31 02:17:45.644088 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.644079    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e6d59f-a66f-4c5c-aff8-8725deabb93f-kube-api-access-hvb76\" (UniqueName: \"kubernetes.io/projected/17e6d59f-a66f-4c5c-aff8-8725deabb93f-kube-api-access-hvb76\") pod \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") "
Oct 31 02:17:45.644345 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.644113    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-workload-socket\") pod \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") "
Oct 31 02:17:45.644345 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.644145    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istiod-ca-cert\") pod \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") "
Oct 31 02:17:45.644345 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.644192    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-credential-socket\") pod \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") "
Oct 31 02:17:45.644345 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.644337    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-data\") pod \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\" (UID: \"17e6d59f-a66f-4c5c-aff8-8725deabb93f\") "
Oct 31 02:17:45.645504 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.644874    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istiod-ca-cert" pod "17e6d59f-a66f-4c5c-aff8-8725deabb93f" (UID: "17e6d59f-a66f-4c5c-aff8-8725deabb93f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:45.645504 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.645102    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-credential-socket" pod "17e6d59f-a66f-4c5c-aff8-8725deabb93f" (UID: "17e6d59f-a66f-4c5c-aff8-8725deabb93f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:45.645504 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.645225    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:45.645504 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.645243    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:45.645806 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.645544    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-data" pod "17e6d59f-a66f-4c5c-aff8-8725deabb93f" (UID: "17e6d59f-a66f-4c5c-aff8-8725deabb93f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:45.645806 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.645598    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-workload-socket" pod "17e6d59f-a66f-4c5c-aff8-8725deabb93f" (UID: "17e6d59f-a66f-4c5c-aff8-8725deabb93f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:45.645988 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.645963    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-workload-certs" pod "17e6d59f-a66f-4c5c-aff8-8725deabb93f" (UID: "17e6d59f-a66f-4c5c-aff8-8725deabb93f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:45.647817 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.647789    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e6d59f-a66f-4c5c-aff8-8725deabb93f-kube-api-access-hvb76" pod "17e6d59f-a66f-4c5c-aff8-8725deabb93f" (UID: "17e6d59f-a66f-4c5c-aff8-8725deabb93f"). InnerVolumeSpecName "kube-api-access-hvb76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:45.647934 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.647920    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-envoy" pod "17e6d59f-a66f-4c5c-aff8-8725deabb93f" (UID: "17e6d59f-a66f-4c5c-aff8-8725deabb93f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:45.648894 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.648868    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-token" pod "17e6d59f-a66f-4c5c-aff8-8725deabb93f" (UID: "17e6d59f-a66f-4c5c-aff8-8725deabb93f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:45.649802 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.649777    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-podinfo" pod "17e6d59f-a66f-4c5c-aff8-8725deabb93f" (UID: "17e6d59f-a66f-4c5c-aff8-8725deabb93f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:17:45.683804 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.683759    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"]
Oct 31 02:17:45.684084 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.684053    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq" podUID="86d5dc42-ef1e-4ce5-9f29-386030d4524b" containerName="istio-proxy" containerID="cri-o://6a7e86081199573084afa4897354324fe9671948e39d167df3500d57e861efce" gracePeriod=30
Oct 31 02:17:45.746906 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.746866    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:45.746906 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.746902    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:45.746906 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.746912    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:45.747158 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.746921    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:45.747158 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.746931    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvb76\" (UniqueName: \"kubernetes.io/projected/17e6d59f-a66f-4c5c-aff8-8725deabb93f-kube-api-access-hvb76\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:45.747158 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.746940    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:45.747158 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:45.746950    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e6d59f-a66f-4c5c-aff8-8725deabb93f-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:46.119813 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:46.119778    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44fe8812-7863-4904-bb34-5ee90f27db75" path="/var/lib/kubelet/pods/44fe8812-7863-4904-bb34-5ee90f27db75/volumes"
Oct 31 02:17:46.251753 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:46.251716    5650 generic.go:356] "Generic (PLEG): container finished" podID="17e6d59f-a66f-4c5c-aff8-8725deabb93f" containerID="d9882cb0bd8d18f1b22605da5a609e49f80bfdf426cbb280e8f9c1369234a920" exitCode=0
Oct 31 02:17:46.251935 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:46.251779    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"
Oct 31 02:17:46.251935 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:46.251795    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m" event={"ID":"17e6d59f-a66f-4c5c-aff8-8725deabb93f","Type":"ContainerDied","Data":"d9882cb0bd8d18f1b22605da5a609e49f80bfdf426cbb280e8f9c1369234a920"}
Oct 31 02:17:46.251935 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:46.251860    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m" event={"ID":"17e6d59f-a66f-4c5c-aff8-8725deabb93f","Type":"ContainerDied","Data":"a29b026738a98da55583be257a70e42514d3102b6ddf322ef1f4ab9a23fa2fa2"}
Oct 31 02:17:46.251935 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:46.251891    5650 scope.go:122] "RemoveContainer" containerID="d9882cb0bd8d18f1b22605da5a609e49f80bfdf426cbb280e8f9c1369234a920"
Oct 31 02:17:46.265063 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:46.265039    5650 scope.go:122] "RemoveContainer" containerID="d01fa1c2f3c95eb99577cfdf4e8a180ad2c60310fd15f68d170fc4b0e15383c2"
Oct 31 02:17:46.276484 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:46.276439    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"]
Oct 31 02:17:46.276708 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:46.276684    5650 scope.go:122] "RemoveContainer" containerID="d9882cb0bd8d18f1b22605da5a609e49f80bfdf426cbb280e8f9c1369234a920"
Oct 31 02:17:46.277095 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:17:46.277062    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9882cb0bd8d18f1b22605da5a609e49f80bfdf426cbb280e8f9c1369234a920\": container with ID starting with d9882cb0bd8d18f1b22605da5a609e49f80bfdf426cbb280e8f9c1369234a920 not found: ID does not exist" containerID="d9882cb0bd8d18f1b22605da5a609e49f80bfdf426cbb280e8f9c1369234a920"
Oct 31 02:17:46.277191 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:46.277106    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9882cb0bd8d18f1b22605da5a609e49f80bfdf426cbb280e8f9c1369234a920"} err="failed to get container status \"d9882cb0bd8d18f1b22605da5a609e49f80bfdf426cbb280e8f9c1369234a920\": rpc error: code = NotFound desc = could not find container \"d9882cb0bd8d18f1b22605da5a609e49f80bfdf426cbb280e8f9c1369234a920\": container with ID starting with d9882cb0bd8d18f1b22605da5a609e49f80bfdf426cbb280e8f9c1369234a920 not found: ID does not exist"
Oct 31 02:17:46.277191 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:46.277132    5650 scope.go:122] "RemoveContainer" containerID="d01fa1c2f3c95eb99577cfdf4e8a180ad2c60310fd15f68d170fc4b0e15383c2"
Oct 31 02:17:46.278938 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:17:46.278908    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d01fa1c2f3c95eb99577cfdf4e8a180ad2c60310fd15f68d170fc4b0e15383c2\": container with ID starting with d01fa1c2f3c95eb99577cfdf4e8a180ad2c60310fd15f68d170fc4b0e15383c2 not found: ID does not exist" containerID="d01fa1c2f3c95eb99577cfdf4e8a180ad2c60310fd15f68d170fc4b0e15383c2"
Oct 31 02:17:46.279030 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:46.278950    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d01fa1c2f3c95eb99577cfdf4e8a180ad2c60310fd15f68d170fc4b0e15383c2"} err="failed to get container status \"d01fa1c2f3c95eb99577cfdf4e8a180ad2c60310fd15f68d170fc4b0e15383c2\": rpc error: code = NotFound desc = could not find container \"d01fa1c2f3c95eb99577cfdf4e8a180ad2c60310fd15f68d170fc4b0e15383c2\": container with ID starting with d01fa1c2f3c95eb99577cfdf4e8a180ad2c60310fd15f68d170fc4b0e15383c2 not found: ID does not exist"
Oct 31 02:17:46.281957 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:46.281936    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5555cb-openshift-default-54f4f8f5c5-4gj4m"]
Oct 31 02:17:46.586925 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:46.586877    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh" podUID="877fff39-e2ad-488e-a14a-60d71bbe02f5" containerName="payload-processing" containerID="cri-o://bc30f0ee9541c90b14e8874827379571f19c10db7cae9ea00b86eb861917c3b5" gracePeriod=25
Oct 31 02:17:46.605305 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:46.605261    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk" podUID="f4a3c58e-ec8b-4dd8-94dd-9c4674840b30" containerName="payload-pre-processing" containerID="cri-o://a47e9d3338d3aa0541aaa177b65c5da550c7c1fa2b68d3f2b1b4b5562cfad2a6" gracePeriod=25
Oct 31 02:17:48.013138 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:48.013099    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh" podUID="877fff39-e2ad-488e-a14a-60d71bbe02f5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:17:48.051396 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:48.051358    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk" podUID="f4a3c58e-ec8b-4dd8-94dd-9c4674840b30" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:17:48.119506 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:48.119464    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17e6d59f-a66f-4c5c-aff8-8725deabb93f" path="/var/lib/kubelet/pods/17e6d59f-a66f-4c5c-aff8-8725deabb93f/volumes"
Oct 31 02:17:50.997830 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:50.997803    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:17:51.020792 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.020761    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6"]
Oct 31 02:17:51.139196 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.139156    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-envoy\") pod \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") "
Oct 31 02:17:51.139196 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.139198    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-token\" (UniqueName: \"kubernetes.io/projected/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-token\") pod \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") "
Oct 31 02:17:51.139548 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.139239    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-workload-socket\") pod \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") "
Oct 31 02:17:51.139548 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.139437    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istiod-ca-cert\") pod \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") "
Oct 31 02:17:51.139682 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.139550    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d5dc42-ef1e-4ce5-9f29-386030d4524b-kube-api-access-jcc6t\" (UniqueName: \"kubernetes.io/projected/86d5dc42-ef1e-4ce5-9f29-386030d4524b-kube-api-access-jcc6t\") pod \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") "
Oct 31 02:17:51.139682 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.139554    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-workload-socket" pod "86d5dc42-ef1e-4ce5-9f29-386030d4524b" (UID: "86d5dc42-ef1e-4ce5-9f29-386030d4524b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:51.139862 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.139841    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istiod-ca-cert" pod "86d5dc42-ef1e-4ce5-9f29-386030d4524b" (UID: "86d5dc42-ef1e-4ce5-9f29-386030d4524b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:51.139922 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.139889    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-data\") pod \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") "
Oct 31 02:17:51.139979 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.139953    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-credential-socket\") pod \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") "
Oct 31 02:17:51.140076 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.140061    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-podinfo\") pod \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") "
Oct 31 02:17:51.140139 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.140099    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-workload-certs\") pod \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\" (UID: \"86d5dc42-ef1e-4ce5-9f29-386030d4524b\") "
Oct 31 02:17:51.140595 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.140539    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-workload-certs" pod "86d5dc42-ef1e-4ce5-9f29-386030d4524b" (UID: "86d5dc42-ef1e-4ce5-9f29-386030d4524b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:51.140595 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.140563    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-credential-socket" pod "86d5dc42-ef1e-4ce5-9f29-386030d4524b" (UID: "86d5dc42-ef1e-4ce5-9f29-386030d4524b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:51.141002 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.140984    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:51.141096 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.141009    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:51.141096 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.141023    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:51.141096 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.141037    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:51.141484 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.141295    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-data" pod "86d5dc42-ef1e-4ce5-9f29-386030d4524b" (UID: "86d5dc42-ef1e-4ce5-9f29-386030d4524b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:51.143224 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.143198    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-token" pod "86d5dc42-ef1e-4ce5-9f29-386030d4524b" (UID: "86d5dc42-ef1e-4ce5-9f29-386030d4524b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:51.143546 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.143521    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d5dc42-ef1e-4ce5-9f29-386030d4524b-kube-api-access-jcc6t" pod "86d5dc42-ef1e-4ce5-9f29-386030d4524b" (UID: "86d5dc42-ef1e-4ce5-9f29-386030d4524b"). InnerVolumeSpecName "kube-api-access-jcc6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:51.144716 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.144689    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-podinfo" pod "86d5dc42-ef1e-4ce5-9f29-386030d4524b" (UID: "86d5dc42-ef1e-4ce5-9f29-386030d4524b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:17:51.144875 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.144853    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-envoy" pod "86d5dc42-ef1e-4ce5-9f29-386030d4524b" (UID: "86d5dc42-ef1e-4ce5-9f29-386030d4524b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:17:51.242803 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.242716    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:51.242803 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.242747    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:51.242803 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.242757    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:51.242803 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.242766    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcc6t\" (UniqueName: \"kubernetes.io/projected/86d5dc42-ef1e-4ce5-9f29-386030d4524b-kube-api-access-jcc6t\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:51.242803 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.242777    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d5dc42-ef1e-4ce5-9f29-386030d4524b-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:51.342787 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.333294    5650 generic.go:356] "Generic (PLEG): container finished" podID="86d5dc42-ef1e-4ce5-9f29-386030d4524b" containerID="6a7e86081199573084afa4897354324fe9671948e39d167df3500d57e861efce" exitCode=0
Oct 31 02:17:51.342787 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.333425    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq" event={"ID":"86d5dc42-ef1e-4ce5-9f29-386030d4524b","Type":"ContainerDied","Data":"6a7e86081199573084afa4897354324fe9671948e39d167df3500d57e861efce"}
Oct 31 02:17:51.342787 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.333526    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq" event={"ID":"86d5dc42-ef1e-4ce5-9f29-386030d4524b","Type":"ContainerDied","Data":"d69c8702e8c36ddd6f966c46dba4fd406fb45062605ef40662fe380aac96d8b1"}
Oct 31 02:17:51.342787 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.333556    5650 scope.go:122] "RemoveContainer" containerID="6a7e86081199573084afa4897354324fe9671948e39d167df3500d57e861efce"
Oct 31 02:17:51.342787 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.333721    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"
Oct 31 02:17:51.378183 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.376159    5650 scope.go:122] "RemoveContainer" containerID="f3bfe1798fd1f9c7b6f9cc2d942897848f4e6ff1772c910fada3fced06f53a6e"
Oct 31 02:17:51.379979 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.379478    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"]
Oct 31 02:17:51.384375 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.384338    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-c088fd-openshift-default-7f9d69c989-9s9gq"]
Oct 31 02:17:51.397742 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.397712    5650 scope.go:122] "RemoveContainer" containerID="6a7e86081199573084afa4897354324fe9671948e39d167df3500d57e861efce"
Oct 31 02:17:51.398084 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:17:51.398063    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a7e86081199573084afa4897354324fe9671948e39d167df3500d57e861efce\": container with ID starting with 6a7e86081199573084afa4897354324fe9671948e39d167df3500d57e861efce not found: ID does not exist" containerID="6a7e86081199573084afa4897354324fe9671948e39d167df3500d57e861efce"
Oct 31 02:17:51.398152 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.398096    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a7e86081199573084afa4897354324fe9671948e39d167df3500d57e861efce"} err="failed to get container status \"6a7e86081199573084afa4897354324fe9671948e39d167df3500d57e861efce\": rpc error: code = NotFound desc = could not find container \"6a7e86081199573084afa4897354324fe9671948e39d167df3500d57e861efce\": container with ID starting with 6a7e86081199573084afa4897354324fe9671948e39d167df3500d57e861efce not found: ID does not exist"
Oct 31 02:17:51.398152 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.398114    5650 scope.go:122] "RemoveContainer" containerID="f3bfe1798fd1f9c7b6f9cc2d942897848f4e6ff1772c910fada3fced06f53a6e"
Oct 31 02:17:51.398428 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:17:51.398407    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3bfe1798fd1f9c7b6f9cc2d942897848f4e6ff1772c910fada3fced06f53a6e\": container with ID starting with f3bfe1798fd1f9c7b6f9cc2d942897848f4e6ff1772c910fada3fced06f53a6e not found: ID does not exist" containerID="f3bfe1798fd1f9c7b6f9cc2d942897848f4e6ff1772c910fada3fced06f53a6e"
Oct 31 02:17:51.398551 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:51.398434    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3bfe1798fd1f9c7b6f9cc2d942897848f4e6ff1772c910fada3fced06f53a6e"} err="failed to get container status \"f3bfe1798fd1f9c7b6f9cc2d942897848f4e6ff1772c910fada3fced06f53a6e\": rpc error: code = NotFound desc = could not find container \"f3bfe1798fd1f9c7b6f9cc2d942897848f4e6ff1772c910fada3fced06f53a6e\": container with ID starting with f3bfe1798fd1f9c7b6f9cc2d942897848f4e6ff1772c910fada3fced06f53a6e not found: ID does not exist"
Oct 31 02:17:52.119237 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:52.119204    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86d5dc42-ef1e-4ce5-9f29-386030d4524b" path="/var/lib/kubelet/pods/86d5dc42-ef1e-4ce5-9f29-386030d4524b/volumes"
Oct 31 02:17:52.726200 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:52.726157    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg" podUID="c24d5105-6144-4a56-8aad-39be92445ea5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:17:52.726200 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:52.726181    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc" podUID="6a1974b0-604c-4805-b965-0b0cfcf89b30" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:17:56.021951 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:56.021859    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6" podUID="816cf08b-0669-44c0-a959-53c3be786f1d" containerName="payload-processing" containerID="cri-o://d9c735d0d8a595b2eaec311e6e16f6a9a48ed23fab362245552e9a01114b399a" gracePeriod=25
Oct 31 02:17:56.334643 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:56.334610    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6"
Oct 31 02:17:56.353404 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:56.353373    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/816cf08b-0669-44c0-a959-53c3be786f1d-kube-api-access-btddh\" (UniqueName: \"kubernetes.io/projected/816cf08b-0669-44c0-a959-53c3be786f1d-kube-api-access-btddh\") pod \"816cf08b-0669-44c0-a959-53c3be786f1d\" (UID: \"816cf08b-0669-44c0-a959-53c3be786f1d\") "
Oct 31 02:17:56.353601 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:56.353585    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/816cf08b-0669-44c0-a959-53c3be786f1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816cf08b-0669-44c0-a959-53c3be786f1d-plugins-config-volume\") pod \"816cf08b-0669-44c0-a959-53c3be786f1d\" (UID: \"816cf08b-0669-44c0-a959-53c3be786f1d\") "
Oct 31 02:17:56.354000 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:56.353969    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816cf08b-0669-44c0-a959-53c3be786f1d-plugins-config-volume" pod "816cf08b-0669-44c0-a959-53c3be786f1d" (UID: "816cf08b-0669-44c0-a959-53c3be786f1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:17:56.354230 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:56.354199    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/816cf08b-0669-44c0-a959-53c3be786f1d-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:56.355635 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:56.355615    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/816cf08b-0669-44c0-a959-53c3be786f1d-kube-api-access-btddh" pod "816cf08b-0669-44c0-a959-53c3be786f1d" (UID: "816cf08b-0669-44c0-a959-53c3be786f1d"). InnerVolumeSpecName "kube-api-access-btddh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:17:56.372948 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:56.372912    5650 generic.go:356] "Generic (PLEG): container finished" podID="816cf08b-0669-44c0-a959-53c3be786f1d" containerID="d9c735d0d8a595b2eaec311e6e16f6a9a48ed23fab362245552e9a01114b399a" exitCode=0
Oct 31 02:17:56.373165 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:56.372964    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6"
Oct 31 02:17:56.373165 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:56.373066    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6" event={"ID":"816cf08b-0669-44c0-a959-53c3be786f1d","Type":"ContainerDied","Data":"d9c735d0d8a595b2eaec311e6e16f6a9a48ed23fab362245552e9a01114b399a"}
Oct 31 02:17:56.373165 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:56.373114    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6" event={"ID":"816cf08b-0669-44c0-a959-53c3be786f1d","Type":"ContainerDied","Data":"f8da060d83675f4da044efb456d445fc836371287436f851ad8d8b357a3deeaa"}
Oct 31 02:17:56.373165 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:56.373142    5650 scope.go:122] "RemoveContainer" containerID="d9c735d0d8a595b2eaec311e6e16f6a9a48ed23fab362245552e9a01114b399a"
Oct 31 02:17:56.395977 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:56.395948    5650 scope.go:122] "RemoveContainer" containerID="d9c735d0d8a595b2eaec311e6e16f6a9a48ed23fab362245552e9a01114b399a"
Oct 31 02:17:56.396411 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:17:56.396388    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9c735d0d8a595b2eaec311e6e16f6a9a48ed23fab362245552e9a01114b399a\": container with ID starting with d9c735d0d8a595b2eaec311e6e16f6a9a48ed23fab362245552e9a01114b399a not found: ID does not exist" containerID="d9c735d0d8a595b2eaec311e6e16f6a9a48ed23fab362245552e9a01114b399a"
Oct 31 02:17:56.396548 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:56.396417    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9c735d0d8a595b2eaec311e6e16f6a9a48ed23fab362245552e9a01114b399a"} err="failed to get container status \"d9c735d0d8a595b2eaec311e6e16f6a9a48ed23fab362245552e9a01114b399a\": rpc error: code = NotFound desc = could not find container \"d9c735d0d8a595b2eaec311e6e16f6a9a48ed23fab362245552e9a01114b399a\": container with ID starting with d9c735d0d8a595b2eaec311e6e16f6a9a48ed23fab362245552e9a01114b399a not found: ID does not exist"
Oct 31 02:17:56.405854 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:56.405814    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6"]
Oct 31 02:17:56.409662 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:56.409634    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-89e955-77fd5f8999-llbs6"]
Oct 31 02:17:56.455085 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:56.455047    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btddh\" (UniqueName: \"kubernetes.io/projected/816cf08b-0669-44c0-a959-53c3be786f1d-kube-api-access-btddh\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:17:56.480721 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:56.480679    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2"]
Oct 31 02:17:56.492394 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:56.492355    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr"]
Oct 31 02:17:58.013405 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:58.013367    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh" podUID="877fff39-e2ad-488e-a14a-60d71bbe02f5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:17:58.051753 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:58.051714    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk" podUID="f4a3c58e-ec8b-4dd8-94dd-9c4674840b30" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:17:58.128374 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:17:58.128324    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="816cf08b-0669-44c0-a959-53c3be786f1d" path="/var/lib/kubelet/pods/816cf08b-0669-44c0-a959-53c3be786f1d/volumes"
Oct 31 02:17:58.554368 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:17:58.554323    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:18:01.481549 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:01.481508    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2" podUID="8ac2ebbd-44b7-499a-be04-4d419002e32f" containerName="payload-processing" containerID="cri-o://1274a09d74b8206892ef582528e6f6de9b37ce25c728938292c30775e2aa1ef3" gracePeriod=25
Oct 31 02:18:01.492897 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:01.492857    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr" podUID="3d3621cc-41bb-4583-b189-ce7402a285cd" containerName="payload-pre-processing" containerID="cri-o://ef5b8aad127c1dff1cc7462a1904495fc686f2e64d18ba9297fd686da26758ff" gracePeriod=25
Oct 31 02:18:01.542976 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:01.542938    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"]
Oct 31 02:18:01.543247 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:01.543216    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn" podUID="4238ae64-8bce-4c8d-8f14-339706f25bed" containerName="istio-proxy" containerID="cri-o://fbd7c9b84063b4d6a7134c102f907a50a2d84e9e76efc2f830d83daad3c36736" gracePeriod=30
Oct 31 02:18:02.329228 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.329200    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc"
Oct 31 02:18:02.371368 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.371337    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg"
Oct 31 02:18:02.417047 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.416996    5650 generic.go:356] "Generic (PLEG): container finished" podID="c24d5105-6144-4a56-8aad-39be92445ea5" containerID="f294c53ed00eb8a48dac3d12ab0b8e3a277f398c670d457cf9e423a9fc7a9d6e" exitCode=0
Oct 31 02:18:02.417047 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.417038    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg"
Oct 31 02:18:02.417330 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.417069    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg" event={"ID":"c24d5105-6144-4a56-8aad-39be92445ea5","Type":"ContainerDied","Data":"f294c53ed00eb8a48dac3d12ab0b8e3a277f398c670d457cf9e423a9fc7a9d6e"}
Oct 31 02:18:02.417330 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.417135    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg" event={"ID":"c24d5105-6144-4a56-8aad-39be92445ea5","Type":"ContainerDied","Data":"14a34086605cbc504c108934564de781a7b40e2f4f932f5ae28168ab6fb2ee88"}
Oct 31 02:18:02.417330 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.417175    5650 scope.go:122] "RemoveContainer" containerID="f294c53ed00eb8a48dac3d12ab0b8e3a277f398c670d457cf9e423a9fc7a9d6e"
Oct 31 02:18:02.418685 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.418661    5650 generic.go:356] "Generic (PLEG): container finished" podID="6a1974b0-604c-4805-b965-0b0cfcf89b30" containerID="2e06fbdd7d5a1688ac2c939b0f5d8c81211fa1f6d10390af57aa348b3c95d1a3" exitCode=0
Oct 31 02:18:02.418811 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.418717    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc"
Oct 31 02:18:02.418811 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.418785    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc" event={"ID":"6a1974b0-604c-4805-b965-0b0cfcf89b30","Type":"ContainerDied","Data":"2e06fbdd7d5a1688ac2c939b0f5d8c81211fa1f6d10390af57aa348b3c95d1a3"}
Oct 31 02:18:02.418904 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.418830    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc" event={"ID":"6a1974b0-604c-4805-b965-0b0cfcf89b30","Type":"ContainerDied","Data":"fbd013d074381a862662770f6ae7a48212f2205c6cac87980b18fc971d4a2083"}
Oct 31 02:18:02.431161 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.431138    5650 scope.go:122] "RemoveContainer" containerID="f294c53ed00eb8a48dac3d12ab0b8e3a277f398c670d457cf9e423a9fc7a9d6e"
Oct 31 02:18:02.431538 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:18:02.431514    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f294c53ed00eb8a48dac3d12ab0b8e3a277f398c670d457cf9e423a9fc7a9d6e\": container with ID starting with f294c53ed00eb8a48dac3d12ab0b8e3a277f398c670d457cf9e423a9fc7a9d6e not found: ID does not exist" containerID="f294c53ed00eb8a48dac3d12ab0b8e3a277f398c670d457cf9e423a9fc7a9d6e"
Oct 31 02:18:02.431595 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.431549    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f294c53ed00eb8a48dac3d12ab0b8e3a277f398c670d457cf9e423a9fc7a9d6e"} err="failed to get container status \"f294c53ed00eb8a48dac3d12ab0b8e3a277f398c670d457cf9e423a9fc7a9d6e\": rpc error: code = NotFound desc = could not find container \"f294c53ed00eb8a48dac3d12ab0b8e3a277f398c670d457cf9e423a9fc7a9d6e\": container with ID starting with f294c53ed00eb8a48dac3d12ab0b8e3a277f398c670d457cf9e423a9fc7a9d6e not found: ID does not exist"
Oct 31 02:18:02.431595 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.431564    5650 scope.go:122] "RemoveContainer" containerID="2e06fbdd7d5a1688ac2c939b0f5d8c81211fa1f6d10390af57aa348b3c95d1a3"
Oct 31 02:18:02.442664 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.442638    5650 scope.go:122] "RemoveContainer" containerID="2e06fbdd7d5a1688ac2c939b0f5d8c81211fa1f6d10390af57aa348b3c95d1a3"
Oct 31 02:18:02.442989 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:18:02.442959    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e06fbdd7d5a1688ac2c939b0f5d8c81211fa1f6d10390af57aa348b3c95d1a3\": container with ID starting with 2e06fbdd7d5a1688ac2c939b0f5d8c81211fa1f6d10390af57aa348b3c95d1a3 not found: ID does not exist" containerID="2e06fbdd7d5a1688ac2c939b0f5d8c81211fa1f6d10390af57aa348b3c95d1a3"
Oct 31 02:18:02.443100 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.442998    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e06fbdd7d5a1688ac2c939b0f5d8c81211fa1f6d10390af57aa348b3c95d1a3"} err="failed to get container status \"2e06fbdd7d5a1688ac2c939b0f5d8c81211fa1f6d10390af57aa348b3c95d1a3\": rpc error: code = NotFound desc = could not find container \"2e06fbdd7d5a1688ac2c939b0f5d8c81211fa1f6d10390af57aa348b3c95d1a3\": container with ID starting with 2e06fbdd7d5a1688ac2c939b0f5d8c81211fa1f6d10390af57aa348b3c95d1a3 not found: ID does not exist"
Oct 31 02:18:02.445438 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.445416    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c24d5105-6144-4a56-8aad-39be92445ea5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24d5105-6144-4a56-8aad-39be92445ea5-plugins-config-volume\") pod \"c24d5105-6144-4a56-8aad-39be92445ea5\" (UID: \"c24d5105-6144-4a56-8aad-39be92445ea5\") "
Oct 31 02:18:02.445567 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.445510    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c24d5105-6144-4a56-8aad-39be92445ea5-kube-api-access-ft5nn\" (UniqueName: \"kubernetes.io/projected/c24d5105-6144-4a56-8aad-39be92445ea5-kube-api-access-ft5nn\") pod \"c24d5105-6144-4a56-8aad-39be92445ea5\" (UID: \"c24d5105-6144-4a56-8aad-39be92445ea5\") "
Oct 31 02:18:02.445632 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.445592    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a1974b0-604c-4805-b965-0b0cfcf89b30-kube-api-access-rbjh4\" (UniqueName: \"kubernetes.io/projected/6a1974b0-604c-4805-b965-0b0cfcf89b30-kube-api-access-rbjh4\") pod \"6a1974b0-604c-4805-b965-0b0cfcf89b30\" (UID: \"6a1974b0-604c-4805-b965-0b0cfcf89b30\") "
Oct 31 02:18:02.445753 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.445735    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a1974b0-604c-4805-b965-0b0cfcf89b30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a1974b0-604c-4805-b965-0b0cfcf89b30-plugins-config-volume\") pod \"6a1974b0-604c-4805-b965-0b0cfcf89b30\" (UID: \"6a1974b0-604c-4805-b965-0b0cfcf89b30\") "
Oct 31 02:18:02.445841 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.445818    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c24d5105-6144-4a56-8aad-39be92445ea5-plugins-config-volume" pod "c24d5105-6144-4a56-8aad-39be92445ea5" (UID: "c24d5105-6144-4a56-8aad-39be92445ea5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:02.446109 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.446085    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a1974b0-604c-4805-b965-0b0cfcf89b30-plugins-config-volume" pod "6a1974b0-604c-4805-b965-0b0cfcf89b30" (UID: "6a1974b0-604c-4805-b965-0b0cfcf89b30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:02.446257 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.446238    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a1974b0-604c-4805-b965-0b0cfcf89b30-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:02.446326 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.446262    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24d5105-6144-4a56-8aad-39be92445ea5-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:02.447621 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.447602    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24d5105-6144-4a56-8aad-39be92445ea5-kube-api-access-ft5nn" pod "c24d5105-6144-4a56-8aad-39be92445ea5" (UID: "c24d5105-6144-4a56-8aad-39be92445ea5"). InnerVolumeSpecName "kube-api-access-ft5nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:02.447717 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.447700    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1974b0-604c-4805-b965-0b0cfcf89b30-kube-api-access-rbjh4" pod "6a1974b0-604c-4805-b965-0b0cfcf89b30" (UID: "6a1974b0-604c-4805-b965-0b0cfcf89b30"). InnerVolumeSpecName "kube-api-access-rbjh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:02.547603 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.547499    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft5nn\" (UniqueName: \"kubernetes.io/projected/c24d5105-6144-4a56-8aad-39be92445ea5-kube-api-access-ft5nn\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:02.547603 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.547535    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbjh4\" (UniqueName: \"kubernetes.io/projected/6a1974b0-604c-4805-b965-0b0cfcf89b30-kube-api-access-rbjh4\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:02.753040 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.753005    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc"]
Oct 31 02:18:02.757741 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.757713    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c088fd-856769599-cg7dc"]
Oct 31 02:18:02.771476 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.771426    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg"]
Oct 31 02:18:02.778817 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:02.778788    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c088fd-68448dfd99-gb6sg"]
Oct 31 02:18:03.670856 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:03.670809    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr" podUID="3d3621cc-41bb-4583-b189-ce7402a285cd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:18:03.670856 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:03.670844    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2" podUID="8ac2ebbd-44b7-499a-be04-4d419002e32f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:18:03.913938 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:03.913898    5650 patch_prober.go:29] interesting pod/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:18:03.914093 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:03.913965    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn" podUID="4238ae64-8bce-4c8d-8f14-339706f25bed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:18:03.994190 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:03.994150    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw"]
Oct 31 02:18:04.120185 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:04.120149    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a1974b0-604c-4805-b965-0b0cfcf89b30" path="/var/lib/kubelet/pods/6a1974b0-604c-4805-b965-0b0cfcf89b30/volumes"
Oct 31 02:18:04.120648 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:04.120633    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c24d5105-6144-4a56-8aad-39be92445ea5" path="/var/lib/kubelet/pods/c24d5105-6144-4a56-8aad-39be92445ea5/volumes"
Oct 31 02:18:06.893113 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:06.893082    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh"
Oct 31 02:18:06.942729 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:06.942698    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk"
Oct 31 02:18:06.946325 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:06.946293    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 02:18:07.013139 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.013109    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-envoy\") pod \"4238ae64-8bce-4c8d-8f14-339706f25bed\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") "
Oct 31 02:18:07.013351 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.013152    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/877fff39-e2ad-488e-a14a-60d71bbe02f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/877fff39-e2ad-488e-a14a-60d71bbe02f5-plugins-config-volume\") pod \"877fff39-e2ad-488e-a14a-60d71bbe02f5\" (UID: \"877fff39-e2ad-488e-a14a-60d71bbe02f5\") "
Oct 31 02:18:07.013351 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.013206    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4a3c58e-ec8b-4dd8-94dd-9c4674840b30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a3c58e-ec8b-4dd8-94dd-9c4674840b30-plugins-config-volume\") pod \"f4a3c58e-ec8b-4dd8-94dd-9c4674840b30\" (UID: \"f4a3c58e-ec8b-4dd8-94dd-9c4674840b30\") "
Oct 31 02:18:07.013351 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.013256    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4238ae64-8bce-4c8d-8f14-339706f25bed-kube-api-access-s5d4w\" (UniqueName: \"kubernetes.io/projected/4238ae64-8bce-4c8d-8f14-339706f25bed-kube-api-access-s5d4w\") pod \"4238ae64-8bce-4c8d-8f14-339706f25bed\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") "
Oct 31 02:18:07.013351 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.013301    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-workload-certs\") pod \"4238ae64-8bce-4c8d-8f14-339706f25bed\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") "
Oct 31 02:18:07.013351 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.013328    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-data\") pod \"4238ae64-8bce-4c8d-8f14-339706f25bed\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") "
Oct 31 02:18:07.013676 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.013415    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4238ae64-8bce-4c8d-8f14-339706f25bed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4238ae64-8bce-4c8d-8f14-339706f25bed-istiod-ca-cert\") pod \"4238ae64-8bce-4c8d-8f14-339706f25bed\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") "
Oct 31 02:18:07.013676 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.013498    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4a3c58e-ec8b-4dd8-94dd-9c4674840b30-kube-api-access-wbh2v\" (UniqueName: \"kubernetes.io/projected/f4a3c58e-ec8b-4dd8-94dd-9c4674840b30-kube-api-access-wbh2v\") pod \"f4a3c58e-ec8b-4dd8-94dd-9c4674840b30\" (UID: \"f4a3c58e-ec8b-4dd8-94dd-9c4674840b30\") "
Oct 31 02:18:07.013676 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.013521    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-podinfo\") pod \"4238ae64-8bce-4c8d-8f14-339706f25bed\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") "
Oct 31 02:18:07.013676 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.013561    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/877fff39-e2ad-488e-a14a-60d71bbe02f5-kube-api-access-fppc6\" (UniqueName: \"kubernetes.io/projected/877fff39-e2ad-488e-a14a-60d71bbe02f5-kube-api-access-fppc6\") pod \"877fff39-e2ad-488e-a14a-60d71bbe02f5\" (UID: \"877fff39-e2ad-488e-a14a-60d71bbe02f5\") "
Oct 31 02:18:07.013676 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.013609    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-workload-socket\") pod \"4238ae64-8bce-4c8d-8f14-339706f25bed\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") "
Oct 31 02:18:07.013676 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.013642    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-token\" (UniqueName: \"kubernetes.io/projected/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-token\") pod \"4238ae64-8bce-4c8d-8f14-339706f25bed\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") "
Oct 31 02:18:07.014058 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.013687    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a3c58e-ec8b-4dd8-94dd-9c4674840b30-plugins-config-volume" pod "f4a3c58e-ec8b-4dd8-94dd-9c4674840b30" (UID: "f4a3c58e-ec8b-4dd8-94dd-9c4674840b30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:07.014058 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.013697    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-credential-socket\") pod \"4238ae64-8bce-4c8d-8f14-339706f25bed\" (UID: \"4238ae64-8bce-4c8d-8f14-339706f25bed\") "
Oct 31 02:18:07.014058 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.013917    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-credential-socket" pod "4238ae64-8bce-4c8d-8f14-339706f25bed" (UID: "4238ae64-8bce-4c8d-8f14-339706f25bed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:07.014222 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.014196    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4238ae64-8bce-4c8d-8f14-339706f25bed-istiod-ca-cert" pod "4238ae64-8bce-4c8d-8f14-339706f25bed" (UID: "4238ae64-8bce-4c8d-8f14-339706f25bed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:07.014404 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.014316    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-workload-certs" pod "4238ae64-8bce-4c8d-8f14-339706f25bed" (UID: "4238ae64-8bce-4c8d-8f14-339706f25bed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:07.014645 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.014622    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-data" pod "4238ae64-8bce-4c8d-8f14-339706f25bed" (UID: "4238ae64-8bce-4c8d-8f14-339706f25bed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:07.014750 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.014726    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-workload-socket" pod "4238ae64-8bce-4c8d-8f14-339706f25bed" (UID: "4238ae64-8bce-4c8d-8f14-339706f25bed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:07.014834 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.014820    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/877fff39-e2ad-488e-a14a-60d71bbe02f5-plugins-config-volume" pod "877fff39-e2ad-488e-a14a-60d71bbe02f5" (UID: "877fff39-e2ad-488e-a14a-60d71bbe02f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:07.015285 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.015249    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/877fff39-e2ad-488e-a14a-60d71bbe02f5-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:07.015285 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.015278    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a3c58e-ec8b-4dd8-94dd-9c4674840b30-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:07.015518 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.015296    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:07.015518 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.015310    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:07.015518 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.015326    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4238ae64-8bce-4c8d-8f14-339706f25bed-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:07.015518 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.015340    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:07.015518 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.015354    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:07.015916 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.015886    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-envoy" pod "4238ae64-8bce-4c8d-8f14-339706f25bed" (UID: "4238ae64-8bce-4c8d-8f14-339706f25bed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:07.016396 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.016364    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a3c58e-ec8b-4dd8-94dd-9c4674840b30-kube-api-access-wbh2v" pod "f4a3c58e-ec8b-4dd8-94dd-9c4674840b30" (UID: "f4a3c58e-ec8b-4dd8-94dd-9c4674840b30"). InnerVolumeSpecName "kube-api-access-wbh2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:07.016648 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.016622    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-token" pod "4238ae64-8bce-4c8d-8f14-339706f25bed" (UID: "4238ae64-8bce-4c8d-8f14-339706f25bed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:07.017175 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.017142    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877fff39-e2ad-488e-a14a-60d71bbe02f5-kube-api-access-fppc6" pod "877fff39-e2ad-488e-a14a-60d71bbe02f5" (UID: "877fff39-e2ad-488e-a14a-60d71bbe02f5"). InnerVolumeSpecName "kube-api-access-fppc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:07.017333 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.017317    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4238ae64-8bce-4c8d-8f14-339706f25bed-kube-api-access-s5d4w" pod "4238ae64-8bce-4c8d-8f14-339706f25bed" (UID: "4238ae64-8bce-4c8d-8f14-339706f25bed"). InnerVolumeSpecName "kube-api-access-s5d4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:07.017837 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.017816    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-podinfo" pod "4238ae64-8bce-4c8d-8f14-339706f25bed" (UID: "4238ae64-8bce-4c8d-8f14-339706f25bed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:18:07.096708 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.096622    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"]
Oct 31 02:18:07.116350 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.116316    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5d4w\" (UniqueName: \"kubernetes.io/projected/4238ae64-8bce-4c8d-8f14-339706f25bed-kube-api-access-s5d4w\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:07.116350 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.116344    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbh2v\" (UniqueName: \"kubernetes.io/projected/f4a3c58e-ec8b-4dd8-94dd-9c4674840b30-kube-api-access-wbh2v\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:07.116350 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.116357    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:07.116606 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.116367    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fppc6\" (UniqueName: \"kubernetes.io/projected/877fff39-e2ad-488e-a14a-60d71bbe02f5-kube-api-access-fppc6\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:07.116606 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.116376    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:07.116606 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.116385    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4238ae64-8bce-4c8d-8f14-339706f25bed-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:07.467210 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.467165    5650 generic.go:356] "Generic (PLEG): container finished" podID="f4a3c58e-ec8b-4dd8-94dd-9c4674840b30" containerID="a47e9d3338d3aa0541aaa177b65c5da550c7c1fa2b68d3f2b1b4b5562cfad2a6" exitCode=0
Oct 31 02:18:07.467210 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.467209    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk"
Oct 31 02:18:07.467491 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.467208    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk" event={"ID":"f4a3c58e-ec8b-4dd8-94dd-9c4674840b30","Type":"ContainerDied","Data":"a47e9d3338d3aa0541aaa177b65c5da550c7c1fa2b68d3f2b1b4b5562cfad2a6"}
Oct 31 02:18:07.467491 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.467270    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk" event={"ID":"f4a3c58e-ec8b-4dd8-94dd-9c4674840b30","Type":"ContainerDied","Data":"0deda906384285b7b95bc4c3645a4efb7595b9a12b1b7e7b6f4450c662425193"}
Oct 31 02:18:07.467491 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.467286    5650 scope.go:122] "RemoveContainer" containerID="a47e9d3338d3aa0541aaa177b65c5da550c7c1fa2b68d3f2b1b4b5562cfad2a6"
Oct 31 02:18:07.469429 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.469403    5650 generic.go:356] "Generic (PLEG): container finished" podID="4238ae64-8bce-4c8d-8f14-339706f25bed" containerID="fbd7c9b84063b4d6a7134c102f907a50a2d84e9e76efc2f830d83daad3c36736" exitCode=0
Oct 31 02:18:07.469611 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.469499    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"
Oct 31 02:18:07.469611 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.469521    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn" event={"ID":"4238ae64-8bce-4c8d-8f14-339706f25bed","Type":"ContainerDied","Data":"fbd7c9b84063b4d6a7134c102f907a50a2d84e9e76efc2f830d83daad3c36736"}
Oct 31 02:18:07.469611 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.469579    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn" event={"ID":"4238ae64-8bce-4c8d-8f14-339706f25bed","Type":"ContainerDied","Data":"ba2af375dda50fcec9c245e668e5afeeb2ae41fd3174182e2fc16689e48230ce"}
Oct 31 02:18:07.471486 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.471429    5650 generic.go:356] "Generic (PLEG): container finished" podID="877fff39-e2ad-488e-a14a-60d71bbe02f5" containerID="bc30f0ee9541c90b14e8874827379571f19c10db7cae9ea00b86eb861917c3b5" exitCode=0
Oct 31 02:18:07.471486 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.471475    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh"
Oct 31 02:18:07.471730 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.471568    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh" event={"ID":"877fff39-e2ad-488e-a14a-60d71bbe02f5","Type":"ContainerDied","Data":"bc30f0ee9541c90b14e8874827379571f19c10db7cae9ea00b86eb861917c3b5"}
Oct 31 02:18:07.471730 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.471622    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh" event={"ID":"877fff39-e2ad-488e-a14a-60d71bbe02f5","Type":"ContainerDied","Data":"aa76147e7b087d5ca6f2b5b2f1d82d3cd9d76025bb47905dc854b6a9d5c1d075"}
Oct 31 02:18:07.481616 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.481594    5650 scope.go:122] "RemoveContainer" containerID="a47e9d3338d3aa0541aaa177b65c5da550c7c1fa2b68d3f2b1b4b5562cfad2a6"
Oct 31 02:18:07.481966 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:18:07.481939    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a47e9d3338d3aa0541aaa177b65c5da550c7c1fa2b68d3f2b1b4b5562cfad2a6\": container with ID starting with a47e9d3338d3aa0541aaa177b65c5da550c7c1fa2b68d3f2b1b4b5562cfad2a6 not found: ID does not exist" containerID="a47e9d3338d3aa0541aaa177b65c5da550c7c1fa2b68d3f2b1b4b5562cfad2a6"
Oct 31 02:18:07.482125 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.481976    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a47e9d3338d3aa0541aaa177b65c5da550c7c1fa2b68d3f2b1b4b5562cfad2a6"} err="failed to get container status \"a47e9d3338d3aa0541aaa177b65c5da550c7c1fa2b68d3f2b1b4b5562cfad2a6\": rpc error: code = NotFound desc = could not find container \"a47e9d3338d3aa0541aaa177b65c5da550c7c1fa2b68d3f2b1b4b5562cfad2a6\": container with ID starting with a47e9d3338d3aa0541aaa177b65c5da550c7c1fa2b68d3f2b1b4b5562cfad2a6 not found: ID does not exist"
Oct 31 02:18:07.482125 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.482002    5650 scope.go:122] "RemoveContainer" containerID="fbd7c9b84063b4d6a7134c102f907a50a2d84e9e76efc2f830d83daad3c36736"
Oct 31 02:18:07.493185 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.493156    5650 scope.go:122] "RemoveContainer" containerID="63b85f5e86c8fd464c93877f608e1e6afb85c30df583e60a3906d48b361ad2c1"
Oct 31 02:18:07.500760 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.500732    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk"]
Oct 31 02:18:07.505325 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.505302    5650 scope.go:122] "RemoveContainer" containerID="fbd7c9b84063b4d6a7134c102f907a50a2d84e9e76efc2f830d83daad3c36736"
Oct 31 02:18:07.506696 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.506635    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ed4dce-6b7c7fc9b7-b4stk"]
Oct 31 02:18:07.510478 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:18:07.510412    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbd7c9b84063b4d6a7134c102f907a50a2d84e9e76efc2f830d83daad3c36736\": container with ID starting with fbd7c9b84063b4d6a7134c102f907a50a2d84e9e76efc2f830d83daad3c36736 not found: ID does not exist" containerID="fbd7c9b84063b4d6a7134c102f907a50a2d84e9e76efc2f830d83daad3c36736"
Oct 31 02:18:07.510679 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.510648    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbd7c9b84063b4d6a7134c102f907a50a2d84e9e76efc2f830d83daad3c36736"} err="failed to get container status \"fbd7c9b84063b4d6a7134c102f907a50a2d84e9e76efc2f830d83daad3c36736\": rpc error: code = NotFound desc = could not find container \"fbd7c9b84063b4d6a7134c102f907a50a2d84e9e76efc2f830d83daad3c36736\": container with ID starting with fbd7c9b84063b4d6a7134c102f907a50a2d84e9e76efc2f830d83daad3c36736 not found: ID does not exist"
Oct 31 02:18:07.510789 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.510684    5650 scope.go:122] "RemoveContainer" containerID="63b85f5e86c8fd464c93877f608e1e6afb85c30df583e60a3906d48b361ad2c1"
Oct 31 02:18:07.511298 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:18:07.511273    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63b85f5e86c8fd464c93877f608e1e6afb85c30df583e60a3906d48b361ad2c1\": container with ID starting with 63b85f5e86c8fd464c93877f608e1e6afb85c30df583e60a3906d48b361ad2c1 not found: ID does not exist" containerID="63b85f5e86c8fd464c93877f608e1e6afb85c30df583e60a3906d48b361ad2c1"
Oct 31 02:18:07.511397 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.511303    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63b85f5e86c8fd464c93877f608e1e6afb85c30df583e60a3906d48b361ad2c1"} err="failed to get container status \"63b85f5e86c8fd464c93877f608e1e6afb85c30df583e60a3906d48b361ad2c1\": rpc error: code = NotFound desc = could not find container \"63b85f5e86c8fd464c93877f608e1e6afb85c30df583e60a3906d48b361ad2c1\": container with ID starting with 63b85f5e86c8fd464c93877f608e1e6afb85c30df583e60a3906d48b361ad2c1 not found: ID does not exist"
Oct 31 02:18:07.511397 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.511325    5650 scope.go:122] "RemoveContainer" containerID="bc30f0ee9541c90b14e8874827379571f19c10db7cae9ea00b86eb861917c3b5"
Oct 31 02:18:07.522159 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.522131    5650 scope.go:122] "RemoveContainer" containerID="bc30f0ee9541c90b14e8874827379571f19c10db7cae9ea00b86eb861917c3b5"
Oct 31 02:18:07.522492 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:18:07.522440    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc30f0ee9541c90b14e8874827379571f19c10db7cae9ea00b86eb861917c3b5\": container with ID starting with bc30f0ee9541c90b14e8874827379571f19c10db7cae9ea00b86eb861917c3b5 not found: ID does not exist" containerID="bc30f0ee9541c90b14e8874827379571f19c10db7cae9ea00b86eb861917c3b5"
Oct 31 02:18:07.522565 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.522503    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc30f0ee9541c90b14e8874827379571f19c10db7cae9ea00b86eb861917c3b5"} err="failed to get container status \"bc30f0ee9541c90b14e8874827379571f19c10db7cae9ea00b86eb861917c3b5\": rpc error: code = NotFound desc = could not find container \"bc30f0ee9541c90b14e8874827379571f19c10db7cae9ea00b86eb861917c3b5\": container with ID starting with bc30f0ee9541c90b14e8874827379571f19c10db7cae9ea00b86eb861917c3b5 not found: ID does not exist"
Oct 31 02:18:07.523516 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.523494    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh"]
Oct 31 02:18:07.531199 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.531168    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ed4dce-6f7544cb79-rbssh"]
Oct 31 02:18:07.546030 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.545985    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"]
Oct 31 02:18:07.550928 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:07.550898    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-89e955-openshift-default-6d68f7989-kq5mn"]
Oct 31 02:18:08.118782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:08.118742    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4238ae64-8bce-4c8d-8f14-339706f25bed" path="/var/lib/kubelet/pods/4238ae64-8bce-4c8d-8f14-339706f25bed/volumes"
Oct 31 02:18:08.119591 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:08.119564    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="877fff39-e2ad-488e-a14a-60d71bbe02f5" path="/var/lib/kubelet/pods/877fff39-e2ad-488e-a14a-60d71bbe02f5/volumes"
Oct 31 02:18:08.120230 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:08.120209    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4a3c58e-ec8b-4dd8-94dd-9c4674840b30" path="/var/lib/kubelet/pods/f4a3c58e-ec8b-4dd8-94dd-9c4674840b30/volumes"
Oct 31 02:18:08.995186 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:08.995143    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw" podUID="459bd3ef-75c0-4b05-baa1-90deddc2d4ab" containerName="payload-processing" containerID="cri-o://3efbf4b96d3c399c60c66962bb1969fa6d8aee9170c3766ae98c95e197b27e54" gracePeriod=25
Oct 31 02:18:09.277708 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:09.277682    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw"
Oct 31 02:18:09.352286 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:09.351552    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459bd3ef-75c0-4b05-baa1-90deddc2d4ab-kube-api-access-4j5pl\" (UniqueName: \"kubernetes.io/projected/459bd3ef-75c0-4b05-baa1-90deddc2d4ab-kube-api-access-4j5pl\") pod \"459bd3ef-75c0-4b05-baa1-90deddc2d4ab\" (UID: \"459bd3ef-75c0-4b05-baa1-90deddc2d4ab\") "
Oct 31 02:18:09.352286 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:09.351717    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/459bd3ef-75c0-4b05-baa1-90deddc2d4ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459bd3ef-75c0-4b05-baa1-90deddc2d4ab-plugins-config-volume\") pod \"459bd3ef-75c0-4b05-baa1-90deddc2d4ab\" (UID: \"459bd3ef-75c0-4b05-baa1-90deddc2d4ab\") "
Oct 31 02:18:09.352808 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:09.352782    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459bd3ef-75c0-4b05-baa1-90deddc2d4ab-plugins-config-volume" pod "459bd3ef-75c0-4b05-baa1-90deddc2d4ab" (UID: "459bd3ef-75c0-4b05-baa1-90deddc2d4ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:09.360561 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:09.357502    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459bd3ef-75c0-4b05-baa1-90deddc2d4ab-kube-api-access-4j5pl" pod "459bd3ef-75c0-4b05-baa1-90deddc2d4ab" (UID: "459bd3ef-75c0-4b05-baa1-90deddc2d4ab"). InnerVolumeSpecName "kube-api-access-4j5pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:09.453958 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:09.453911    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j5pl\" (UniqueName: \"kubernetes.io/projected/459bd3ef-75c0-4b05-baa1-90deddc2d4ab-kube-api-access-4j5pl\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:09.453958 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:09.453956    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459bd3ef-75c0-4b05-baa1-90deddc2d4ab-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:09.488485 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:09.488434    5650 generic.go:356] "Generic (PLEG): container finished" podID="459bd3ef-75c0-4b05-baa1-90deddc2d4ab" containerID="3efbf4b96d3c399c60c66962bb1969fa6d8aee9170c3766ae98c95e197b27e54" exitCode=0
Oct 31 02:18:09.488658 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:09.488492    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw" event={"ID":"459bd3ef-75c0-4b05-baa1-90deddc2d4ab","Type":"ContainerDied","Data":"3efbf4b96d3c399c60c66962bb1969fa6d8aee9170c3766ae98c95e197b27e54"}
Oct 31 02:18:09.488658 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:09.488549    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw" event={"ID":"459bd3ef-75c0-4b05-baa1-90deddc2d4ab","Type":"ContainerDied","Data":"1784610b234d3ff2984b5eef9405551da23051d523e4593a99f1e7839bdbb732"}
Oct 31 02:18:09.488658 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:09.488572    5650 scope.go:122] "RemoveContainer" containerID="3efbf4b96d3c399c60c66962bb1969fa6d8aee9170c3766ae98c95e197b27e54"
Oct 31 02:18:09.488658 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:09.488501    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw"
Oct 31 02:18:09.499710 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:09.499689    5650 scope.go:122] "RemoveContainer" containerID="3efbf4b96d3c399c60c66962bb1969fa6d8aee9170c3766ae98c95e197b27e54"
Oct 31 02:18:09.500014 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:18:09.499992    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3efbf4b96d3c399c60c66962bb1969fa6d8aee9170c3766ae98c95e197b27e54\": container with ID starting with 3efbf4b96d3c399c60c66962bb1969fa6d8aee9170c3766ae98c95e197b27e54 not found: ID does not exist" containerID="3efbf4b96d3c399c60c66962bb1969fa6d8aee9170c3766ae98c95e197b27e54"
Oct 31 02:18:09.500057 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:09.500024    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3efbf4b96d3c399c60c66962bb1969fa6d8aee9170c3766ae98c95e197b27e54"} err="failed to get container status \"3efbf4b96d3c399c60c66962bb1969fa6d8aee9170c3766ae98c95e197b27e54\": rpc error: code = NotFound desc = could not find container \"3efbf4b96d3c399c60c66962bb1969fa6d8aee9170c3766ae98c95e197b27e54\": container with ID starting with 3efbf4b96d3c399c60c66962bb1969fa6d8aee9170c3766ae98c95e197b27e54 not found: ID does not exist"
Oct 31 02:18:09.517940 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:09.517851    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw"]
Oct 31 02:18:09.526974 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:09.526944    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-2e6bfd-86bbf5f8c7-zj4nw"]
Oct 31 02:18:10.119163 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:10.119129    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="459bd3ef-75c0-4b05-baa1-90deddc2d4ab" path="/var/lib/kubelet/pods/459bd3ef-75c0-4b05-baa1-90deddc2d4ab/volumes"
Oct 31 02:18:13.670857 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:13.670815    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2" podUID="8ac2ebbd-44b7-499a-be04-4d419002e32f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:18:13.671336 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:13.670872    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr" podUID="3d3621cc-41bb-4583-b189-ce7402a285cd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:18:15.836361 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.836320    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b2209c2-rp8kf"]
Oct 31 02:18:15.836910 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.836887    5650 cpu_manager.go:414] "containerMap: removing container" podUID="c060bf51-21db-4f5b-889d-e655338a9bfa" containerName="istio-proxy"
Oct 31 02:18:15.836910 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.836909    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c060bf51-21db-4f5b-889d-e655338a9bfa" containerName="istio-proxy"
Oct 31 02:18:15.837083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.836926    5650 cpu_manager.go:414] "containerMap: removing container" podUID="877fff39-e2ad-488e-a14a-60d71bbe02f5" containerName="payload-processing"
Oct 31 02:18:15.837083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.836931    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="877fff39-e2ad-488e-a14a-60d71bbe02f5" containerName="payload-processing"
Oct 31 02:18:15.837083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.836942    5650 cpu_manager.go:414] "containerMap: removing container" podUID="17e6d59f-a66f-4c5c-aff8-8725deabb93f" containerName="istio-proxy"
Oct 31 02:18:15.837083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.836947    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e6d59f-a66f-4c5c-aff8-8725deabb93f" containerName="istio-proxy"
Oct 31 02:18:15.837083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.836958    5650 cpu_manager.go:414] "containerMap: removing container" podUID="17e6d59f-a66f-4c5c-aff8-8725deabb93f" containerName="istio-proxy"
Oct 31 02:18:15.837083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.836963    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e6d59f-a66f-4c5c-aff8-8725deabb93f" containerName="istio-proxy"
Oct 31 02:18:15.837083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.836973    5650 cpu_manager.go:414] "containerMap: removing container" podUID="f6eae8ee-0ae3-4beb-9777-e63e8853a9cb" containerName="payload-processing"
Oct 31 02:18:15.837083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.836981    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6eae8ee-0ae3-4beb-9777-e63e8853a9cb" containerName="payload-processing"
Oct 31 02:18:15.837083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.836992    5650 cpu_manager.go:414] "containerMap: removing container" podUID="5edd74fe-6362-4479-b949-c44183eb2b0d" containerName="istio-proxy"
Oct 31 02:18:15.837083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.836998    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5edd74fe-6362-4479-b949-c44183eb2b0d" containerName="istio-proxy"
Oct 31 02:18:15.837083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837008    5650 cpu_manager.go:414] "containerMap: removing container" podUID="d7cacf9b-ad38-4f4b-a68c-f331dc576304" containerName="istio-proxy"
Oct 31 02:18:15.837083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837013    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7cacf9b-ad38-4f4b-a68c-f331dc576304" containerName="istio-proxy"
Oct 31 02:18:15.837083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837020    5650 cpu_manager.go:414] "containerMap: removing container" podUID="9fc48192-d1e7-4f3d-8285-3c0d98465241" containerName="payload-pre-processing"
Oct 31 02:18:15.837083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837026    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc48192-d1e7-4f3d-8285-3c0d98465241" containerName="payload-pre-processing"
Oct 31 02:18:15.837083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837041    5650 cpu_manager.go:414] "containerMap: removing container" podUID="e22feaff-a34e-4be5-b480-30b9db5c5d58" containerName="istio-proxy"
Oct 31 02:18:15.837083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837046    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e22feaff-a34e-4be5-b480-30b9db5c5d58" containerName="istio-proxy"
Oct 31 02:18:15.837083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837054    5650 cpu_manager.go:414] "containerMap: removing container" podUID="86d5dc42-ef1e-4ce5-9f29-386030d4524b" containerName="istio-proxy"
Oct 31 02:18:15.837083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837058    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86d5dc42-ef1e-4ce5-9f29-386030d4524b" containerName="istio-proxy"
Oct 31 02:18:15.837083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837067    5650 cpu_manager.go:414] "containerMap: removing container" podUID="4238ae64-8bce-4c8d-8f14-339706f25bed" containerName="istio-proxy"
Oct 31 02:18:15.837083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837071    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4238ae64-8bce-4c8d-8f14-339706f25bed" containerName="istio-proxy"
Oct 31 02:18:15.837083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837079    5650 cpu_manager.go:414] "containerMap: removing container" podUID="f4a3c58e-ec8b-4dd8-94dd-9c4674840b30" containerName="payload-pre-processing"
Oct 31 02:18:15.837083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837084    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4a3c58e-ec8b-4dd8-94dd-9c4674840b30" containerName="payload-pre-processing"
Oct 31 02:18:15.837083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837091    5650 cpu_manager.go:414] "containerMap: removing container" podUID="816cf08b-0669-44c0-a959-53c3be786f1d" containerName="payload-processing"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837096    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="816cf08b-0669-44c0-a959-53c3be786f1d" containerName="payload-processing"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837107    5650 cpu_manager.go:414] "containerMap: removing container" podUID="6a1974b0-604c-4805-b965-0b0cfcf89b30" containerName="payload-processing"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837112    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a1974b0-604c-4805-b965-0b0cfcf89b30" containerName="payload-processing"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837122    5650 cpu_manager.go:414] "containerMap: removing container" podUID="459bd3ef-75c0-4b05-baa1-90deddc2d4ab" containerName="payload-processing"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837127    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="459bd3ef-75c0-4b05-baa1-90deddc2d4ab" containerName="payload-processing"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837136    5650 cpu_manager.go:414] "containerMap: removing container" podUID="4238ae64-8bce-4c8d-8f14-339706f25bed" containerName="istio-proxy"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837140    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4238ae64-8bce-4c8d-8f14-339706f25bed" containerName="istio-proxy"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837146    5650 cpu_manager.go:414] "containerMap: removing container" podUID="86d5dc42-ef1e-4ce5-9f29-386030d4524b" containerName="istio-proxy"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837150    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86d5dc42-ef1e-4ce5-9f29-386030d4524b" containerName="istio-proxy"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837158    5650 cpu_manager.go:414] "containerMap: removing container" podUID="822c5afc-7c59-422d-a24d-d1b682127173" containerName="payload-pre-processing"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837169    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="822c5afc-7c59-422d-a24d-d1b682127173" containerName="payload-pre-processing"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837180    5650 cpu_manager.go:414] "containerMap: removing container" podUID="218ba464-377e-48e6-a904-59aac3818f28" containerName="payload-processing"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837185    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="218ba464-377e-48e6-a904-59aac3818f28" containerName="payload-processing"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837192    5650 cpu_manager.go:414] "containerMap: removing container" podUID="68500858-f749-44a3-95d1-ebb88191c582" containerName="revoke-keys"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837197    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68500858-f749-44a3-95d1-ebb88191c582" containerName="revoke-keys"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837204    5650 cpu_manager.go:414] "containerMap: removing container" podUID="44fe8812-7863-4904-bb34-5ee90f27db75" containerName="payload-pre-processing"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837208    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44fe8812-7863-4904-bb34-5ee90f27db75" containerName="payload-pre-processing"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837216    5650 cpu_manager.go:414] "containerMap: removing container" podUID="c24d5105-6144-4a56-8aad-39be92445ea5" containerName="payload-pre-processing"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837221    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c24d5105-6144-4a56-8aad-39be92445ea5" containerName="payload-pre-processing"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837288    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4a3c58e-ec8b-4dd8-94dd-9c4674840b30" containerName="payload-pre-processing"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837295    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="816cf08b-0669-44c0-a959-53c3be786f1d" containerName="payload-processing"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837301    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a1974b0-604c-4805-b965-0b0cfcf89b30" containerName="payload-processing"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837307    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="c060bf51-21db-4f5b-889d-e655338a9bfa" containerName="istio-proxy"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837314    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="459bd3ef-75c0-4b05-baa1-90deddc2d4ab" containerName="payload-processing"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837322    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="4238ae64-8bce-4c8d-8f14-339706f25bed" containerName="istio-proxy"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837326    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="86d5dc42-ef1e-4ce5-9f29-386030d4524b" containerName="istio-proxy"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837332    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="822c5afc-7c59-422d-a24d-d1b682127173" containerName="payload-pre-processing"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837336    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="218ba464-377e-48e6-a904-59aac3818f28" containerName="payload-processing"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837342    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="68500858-f749-44a3-95d1-ebb88191c582" containerName="revoke-keys"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837348    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="44fe8812-7863-4904-bb34-5ee90f27db75" containerName="payload-pre-processing"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837353    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="c24d5105-6144-4a56-8aad-39be92445ea5" containerName="payload-pre-processing"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837361    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="877fff39-e2ad-488e-a14a-60d71bbe02f5" containerName="payload-processing"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837367    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e6d59f-a66f-4c5c-aff8-8725deabb93f" containerName="istio-proxy"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837373    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e6d59f-a66f-4c5c-aff8-8725deabb93f" containerName="istio-proxy"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837377    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6eae8ee-0ae3-4beb-9777-e63e8853a9cb" containerName="payload-processing"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837383    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="5edd74fe-6362-4479-b949-c44183eb2b0d" containerName="istio-proxy"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837389    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7cacf9b-ad38-4f4b-a68c-f331dc576304" containerName="istio-proxy"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837395    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc48192-d1e7-4f3d-8285-3c0d98465241" containerName="payload-pre-processing"
Oct 31 02:18:15.837835 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.837401    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="e22feaff-a34e-4be5-b480-30b9db5c5d58" containerName="istio-proxy"
Oct 31 02:18:15.842392 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.842371    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2209c2-rp8kf"
Oct 31 02:18:15.849961 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.849933    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:18:15.850308 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.850285    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b2209c2-rp8kf"]
Oct 31 02:18:15.932945 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.932906    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l"]
Oct 31 02:18:15.943534 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.943498    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1013a448-811b-4c57-ad4a-acb5d91553e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b2209c2-rp8kf\" (UID: \"1013a448-811b-4c57-ad4a-acb5d91553e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2209c2-rp8kf"
Oct 31 02:18:15.943701 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.943562    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp8ht\" (UniqueName: \"kubernetes.io/projected/1013a448-811b-4c57-ad4a-acb5d91553e4-kube-api-access-kp8ht\") pod \"maas-api-revoke-sub-5b2209c2-rp8kf\" (UID: \"1013a448-811b-4c57-ad4a-acb5d91553e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2209c2-rp8kf"
Oct 31 02:18:15.943701 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.943667    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1013a448-811b-4c57-ad4a-acb5d91553e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b2209c2-rp8kf\" (UID: \"1013a448-811b-4c57-ad4a-acb5d91553e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2209c2-rp8kf"
Oct 31 02:18:15.947343 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:15.947318    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:18:16.044938 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:16.044907    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1013a448-811b-4c57-ad4a-acb5d91553e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b2209c2-rp8kf\" (UID: \"1013a448-811b-4c57-ad4a-acb5d91553e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2209c2-rp8kf"
Oct 31 02:18:16.045120 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:16.044959    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp8ht\" (UniqueName: \"kubernetes.io/projected/1013a448-811b-4c57-ad4a-acb5d91553e4-kube-api-access-kp8ht\") pod \"maas-api-revoke-sub-5b2209c2-rp8kf\" (UID: \"1013a448-811b-4c57-ad4a-acb5d91553e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2209c2-rp8kf"
Oct 31 02:18:16.045120 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:16.044984    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1013a448-811b-4c57-ad4a-acb5d91553e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b2209c2-rp8kf\" (UID: \"1013a448-811b-4c57-ad4a-acb5d91553e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2209c2-rp8kf"
Oct 31 02:18:16.045739 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:16.045715    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1013a448-811b-4c57-ad4a-acb5d91553e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b2209c2-rp8kf\" (UID: \"1013a448-811b-4c57-ad4a-acb5d91553e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2209c2-rp8kf"
Oct 31 02:18:16.047374 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:16.047351    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1013a448-811b-4c57-ad4a-acb5d91553e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b2209c2-rp8kf\" (UID: \"1013a448-811b-4c57-ad4a-acb5d91553e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2209c2-rp8kf"
Oct 31 02:18:16.058260 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:16.058233    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp8ht\" (UniqueName: \"kubernetes.io/projected/1013a448-811b-4c57-ad4a-acb5d91553e4-kube-api-access-kp8ht\") pod \"maas-api-revoke-sub-5b2209c2-rp8kf\" (UID: \"1013a448-811b-4c57-ad4a-acb5d91553e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2209c2-rp8kf"
Oct 31 02:18:16.160799 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:16.160718    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2209c2-rp8kf"
Oct 31 02:18:16.571659 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:16.571630    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b2209c2-rp8kf"]
Oct 31 02:18:16.573985 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:18:16.573954    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1013a448_811b_4c57_ad4a_acb5d91553e4.slice/crio-9e936c92fec3e402a3a3c27c919f8c3d9a9c53534fd3a4e74c6c3c9945428fa4 WatchSource:0}: Error finding container 9e936c92fec3e402a3a3c27c919f8c3d9a9c53534fd3a4e74c6c3c9945428fa4: Status 404 returned error can't find the container with id 9e936c92fec3e402a3a3c27c919f8c3d9a9c53534fd3a4e74c6c3c9945428fa4
Oct 31 02:18:17.560212 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:17.560165    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2209c2-rp8kf" event={"ID":"1013a448-811b-4c57-ad4a-acb5d91553e4","Type":"ContainerStarted","Data":"5bfc2512052aaa7e7e8b8d3c7494326bb1612f021849de56bcb1c02aba218837"}
Oct 31 02:18:17.560667 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:17.560264    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2209c2-rp8kf" event={"ID":"1013a448-811b-4c57-ad4a-acb5d91553e4","Type":"ContainerStarted","Data":"9e936c92fec3e402a3a3c27c919f8c3d9a9c53534fd3a4e74c6c3c9945428fa4"}
Oct 31 02:18:17.587933 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:17.587873    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2209c2-rp8kf" podStartSLOduration=2.58785899 podStartE2EDuration="2.58785899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:18:17.583287589 +0000 UTC m=+3453.587001881" watchObservedRunningTime="2026-10-31 02:18:17.58785899 +0000 UTC m=+3453.591573281"
Oct 31 02:18:18.573868 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:18.573826    5650 generic.go:356] "Generic (PLEG): container finished" podID="1013a448-811b-4c57-ad4a-acb5d91553e4" containerID="5bfc2512052aaa7e7e8b8d3c7494326bb1612f021849de56bcb1c02aba218837" exitCode=0
Oct 31 02:18:18.574417 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:18.573997    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2209c2-rp8kf" event={"ID":"1013a448-811b-4c57-ad4a-acb5d91553e4","Type":"ContainerDied","Data":"5bfc2512052aaa7e7e8b8d3c7494326bb1612f021849de56bcb1c02aba218837"}
Oct 31 02:18:18.791782 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:18.791745    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"]
Oct 31 02:18:18.796285 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:18.796256    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"]
Oct 31 02:18:18.814879 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:18.814842    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"]
Oct 31 02:18:19.743055 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:19.743023    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2209c2-rp8kf"
Oct 31 02:18:19.810222 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:19.810188    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1013a448-811b-4c57-ad4a-acb5d91553e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1013a448-811b-4c57-ad4a-acb5d91553e4-maas-api-openshift-ca\") pod \"1013a448-811b-4c57-ad4a-acb5d91553e4\" (UID: \"1013a448-811b-4c57-ad4a-acb5d91553e4\") "
Oct 31 02:18:19.810407 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:19.810231    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1013a448-811b-4c57-ad4a-acb5d91553e4-kube-api-access-kp8ht\" (UniqueName: \"kubernetes.io/projected/1013a448-811b-4c57-ad4a-acb5d91553e4-kube-api-access-kp8ht\") pod \"1013a448-811b-4c57-ad4a-acb5d91553e4\" (UID: \"1013a448-811b-4c57-ad4a-acb5d91553e4\") "
Oct 31 02:18:19.810407 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:19.810397    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1013a448-811b-4c57-ad4a-acb5d91553e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1013a448-811b-4c57-ad4a-acb5d91553e4-maas-api-xks-ca\") pod \"1013a448-811b-4c57-ad4a-acb5d91553e4\" (UID: \"1013a448-811b-4c57-ad4a-acb5d91553e4\") "
Oct 31 02:18:19.810652 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:19.810626    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1013a448-811b-4c57-ad4a-acb5d91553e4-maas-api-openshift-ca" pod "1013a448-811b-4c57-ad4a-acb5d91553e4" (UID: "1013a448-811b-4c57-ad4a-acb5d91553e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:19.810910 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:19.810896    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1013a448-811b-4c57-ad4a-acb5d91553e4-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:19.812464 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:19.812429    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1013a448-811b-4c57-ad4a-acb5d91553e4-maas-api-xks-ca" pod "1013a448-811b-4c57-ad4a-acb5d91553e4" (UID: "1013a448-811b-4c57-ad4a-acb5d91553e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:18:19.812552 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:19.812531    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1013a448-811b-4c57-ad4a-acb5d91553e4-kube-api-access-kp8ht" pod "1013a448-811b-4c57-ad4a-acb5d91553e4" (UID: "1013a448-811b-4c57-ad4a-acb5d91553e4"). InnerVolumeSpecName "kube-api-access-kp8ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:19.912315 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:19.912221    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp8ht\" (UniqueName: \"kubernetes.io/projected/1013a448-811b-4c57-ad4a-acb5d91553e4-kube-api-access-kp8ht\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:19.912315 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:19.912255    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1013a448-811b-4c57-ad4a-acb5d91553e4-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:20.588463 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:20.588419    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2209c2-rp8kf"
Oct 31 02:18:20.588657 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:20.588460    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b2209c2-rp8kf" event={"ID":"1013a448-811b-4c57-ad4a-acb5d91553e4","Type":"ContainerDied","Data":"9e936c92fec3e402a3a3c27c919f8c3d9a9c53534fd3a4e74c6c3c9945428fa4"}
Oct 31 02:18:20.588657 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:20.588517    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e936c92fec3e402a3a3c27c919f8c3d9a9c53534fd3a4e74c6c3c9945428fa4"
Oct 31 02:18:20.934024 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:20.933926    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l" podUID="8969a12d-73b4-4d52-a144-918c72ded4bc" containerName="payload-pre-processing" containerID="cri-o://15ac7aeaaeaadde58c127eb460dccae3f1b72473c0f6086f85769ba1a5e9d298" gracePeriod=25
Oct 31 02:18:21.598864 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:21.598836    5650 generic.go:356] "Generic (PLEG): container finished" podID="8ac2ebbd-44b7-499a-be04-4d419002e32f" containerID="1274a09d74b8206892ef582528e6f6de9b37ce25c728938292c30775e2aa1ef3" exitCode=0
Oct 31 02:18:21.598988 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:21.598915    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2" event={"ID":"8ac2ebbd-44b7-499a-be04-4d419002e32f","Type":"ContainerDied","Data":"1274a09d74b8206892ef582528e6f6de9b37ce25c728938292c30775e2aa1ef3"}
Oct 31 02:18:21.777492 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:21.774570    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2"
Oct 31 02:18:21.837753 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:21.837724    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ac2ebbd-44b7-499a-be04-4d419002e32f-kube-api-access-dlq47\" (UniqueName: \"kubernetes.io/projected/8ac2ebbd-44b7-499a-be04-4d419002e32f-kube-api-access-dlq47\") pod \"8ac2ebbd-44b7-499a-be04-4d419002e32f\" (UID: \"8ac2ebbd-44b7-499a-be04-4d419002e32f\") "
Oct 31 02:18:21.838008 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:21.837790    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ac2ebbd-44b7-499a-be04-4d419002e32f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac2ebbd-44b7-499a-be04-4d419002e32f-plugins-config-volume\") pod \"8ac2ebbd-44b7-499a-be04-4d419002e32f\" (UID: \"8ac2ebbd-44b7-499a-be04-4d419002e32f\") "
Oct 31 02:18:21.838308 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:21.838236    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac2ebbd-44b7-499a-be04-4d419002e32f-plugins-config-volume" pod "8ac2ebbd-44b7-499a-be04-4d419002e32f" (UID: "8ac2ebbd-44b7-499a-be04-4d419002e32f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:21.838521 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:21.838503    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac2ebbd-44b7-499a-be04-4d419002e32f-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:21.842953 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:21.842923    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac2ebbd-44b7-499a-be04-4d419002e32f-kube-api-access-dlq47" pod "8ac2ebbd-44b7-499a-be04-4d419002e32f" (UID: "8ac2ebbd-44b7-499a-be04-4d419002e32f"). InnerVolumeSpecName "kube-api-access-dlq47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:21.843631 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:21.843604    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr"
Oct 31 02:18:21.938970 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:21.938926    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d3621cc-41bb-4583-b189-ce7402a285cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3621cc-41bb-4583-b189-ce7402a285cd-plugins-config-volume\") pod \"3d3621cc-41bb-4583-b189-ce7402a285cd\" (UID: \"3d3621cc-41bb-4583-b189-ce7402a285cd\") "
Oct 31 02:18:21.939472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:21.939095    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3621cc-41bb-4583-b189-ce7402a285cd-kube-api-access-jw47z\" (UniqueName: \"kubernetes.io/projected/3d3621cc-41bb-4583-b189-ce7402a285cd-kube-api-access-jw47z\") pod \"3d3621cc-41bb-4583-b189-ce7402a285cd\" (UID: \"3d3621cc-41bb-4583-b189-ce7402a285cd\") "
Oct 31 02:18:21.939472 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:21.939354    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d3621cc-41bb-4583-b189-ce7402a285cd-plugins-config-volume" pod "3d3621cc-41bb-4583-b189-ce7402a285cd" (UID: "3d3621cc-41bb-4583-b189-ce7402a285cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:21.939742 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:21.939718    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlq47\" (UniqueName: \"kubernetes.io/projected/8ac2ebbd-44b7-499a-be04-4d419002e32f-kube-api-access-dlq47\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:21.939829 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:21.939747    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3621cc-41bb-4583-b189-ce7402a285cd-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:21.941510 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:21.941482    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3621cc-41bb-4583-b189-ce7402a285cd-kube-api-access-jw47z" pod "3d3621cc-41bb-4583-b189-ce7402a285cd" (UID: "3d3621cc-41bb-4583-b189-ce7402a285cd"). InnerVolumeSpecName "kube-api-access-jw47z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:22.041297 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.041219    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw47z\" (UniqueName: \"kubernetes.io/projected/3d3621cc-41bb-4583-b189-ce7402a285cd-kube-api-access-jw47z\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:22.122024 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.121976    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" containerID="cri-o://1fe59413b6d15bb32c99320054d153490ebe26888efe9812e3e4162b31b5eeb3" gracePeriod=45
Oct 31 02:18:22.180293 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.180250    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": dial tcp 10.134.0.60:8000: connect: connection refused"
Oct 31 02:18:22.390230 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.390199    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 02:18:22.447609 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.447559    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-dshm\" (UniqueName: \"kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-dshm\") pod \"176ca180-029d-4252-9ce9-a73297ffb286\" (UID: \"176ca180-029d-4252-9ce9-a73297ffb286\") "
Oct 31 02:18:22.447801 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.447712    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/176ca180-029d-4252-9ce9-a73297ffb286-tls-certs\" (UniqueName: \"kubernetes.io/secret/176ca180-029d-4252-9ce9-a73297ffb286-tls-certs\") pod \"176ca180-029d-4252-9ce9-a73297ffb286\" (UID: \"176ca180-029d-4252-9ce9-a73297ffb286\") "
Oct 31 02:18:22.447801 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.447751    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-tmp-dir\") pod \"176ca180-029d-4252-9ce9-a73297ffb286\" (UID: \"176ca180-029d-4252-9ce9-a73297ffb286\") "
Oct 31 02:18:22.447947 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.447840    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-home\" (UniqueName: \"kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-home\") pod \"176ca180-029d-4252-9ce9-a73297ffb286\" (UID: \"176ca180-029d-4252-9ce9-a73297ffb286\") "
Oct 31 02:18:22.448037 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.448017    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/176ca180-029d-4252-9ce9-a73297ffb286-kube-api-access-4ql72\" (UniqueName: \"kubernetes.io/projected/176ca180-029d-4252-9ce9-a73297ffb286-kube-api-access-4ql72\") pod \"176ca180-029d-4252-9ce9-a73297ffb286\" (UID: \"176ca180-029d-4252-9ce9-a73297ffb286\") "
Oct 31 02:18:22.448108 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.448063    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-model-cache\") pod \"176ca180-029d-4252-9ce9-a73297ffb286\" (UID: \"176ca180-029d-4252-9ce9-a73297ffb286\") "
Oct 31 02:18:22.449177 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.449028    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-home" pod "176ca180-029d-4252-9ce9-a73297ffb286" (UID: "176ca180-029d-4252-9ce9-a73297ffb286"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:22.449388 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.449213    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-tmp-dir" pod "176ca180-029d-4252-9ce9-a73297ffb286" (UID: "176ca180-029d-4252-9ce9-a73297ffb286"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:22.452028 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.451978    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176ca180-029d-4252-9ce9-a73297ffb286-tls-certs" pod "176ca180-029d-4252-9ce9-a73297ffb286" (UID: "176ca180-029d-4252-9ce9-a73297ffb286"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:18:22.452263 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.448946    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-model-cache" pod "176ca180-029d-4252-9ce9-a73297ffb286" (UID: "176ca180-029d-4252-9ce9-a73297ffb286"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:22.456265 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.455128    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176ca180-029d-4252-9ce9-a73297ffb286-kube-api-access-4ql72" pod "176ca180-029d-4252-9ce9-a73297ffb286" (UID: "176ca180-029d-4252-9ce9-a73297ffb286"). InnerVolumeSpecName "kube-api-access-4ql72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:22.456265 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.455904    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-dshm" pod "176ca180-029d-4252-9ce9-a73297ffb286" (UID: "176ca180-029d-4252-9ce9-a73297ffb286"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:22.550417 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.550224    5650 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-home\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:22.550417 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.550272    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ql72\" (UniqueName: \"kubernetes.io/projected/176ca180-029d-4252-9ce9-a73297ffb286-kube-api-access-4ql72\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:22.550417 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.550293    5650 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-model-cache\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:22.550417 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.550308    5650 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-dshm\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:22.550417 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.550322    5650 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176ca180-029d-4252-9ce9-a73297ffb286-tls-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:22.550417 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.550335    5650 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176ca180-029d-4252-9ce9-a73297ffb286-tmp-dir\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:22.609924 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.609882    5650 generic.go:356] "Generic (PLEG): container finished" podID="3d3621cc-41bb-4583-b189-ce7402a285cd" containerID="ef5b8aad127c1dff1cc7462a1904495fc686f2e64d18ba9297fd686da26758ff" exitCode=0
Oct 31 02:18:22.610131 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.609938    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr"
Oct 31 02:18:22.610131 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.609959    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr" event={"ID":"3d3621cc-41bb-4583-b189-ce7402a285cd","Type":"ContainerDied","Data":"ef5b8aad127c1dff1cc7462a1904495fc686f2e64d18ba9297fd686da26758ff"}
Oct 31 02:18:22.610131 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.610043    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr" event={"ID":"3d3621cc-41bb-4583-b189-ce7402a285cd","Type":"ContainerDied","Data":"b736644c6e7ac49868ffd1ce2da5231acd11056b58995933921ba7a8a4eba87c"}
Oct 31 02:18:22.610131 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.610068    5650 scope.go:122] "RemoveContainer" containerID="ef5b8aad127c1dff1cc7462a1904495fc686f2e64d18ba9297fd686da26758ff"
Oct 31 02:18:22.612052 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.612027    5650 generic.go:356] "Generic (PLEG): container finished" podID="176ca180-029d-4252-9ce9-a73297ffb286" containerID="1fe59413b6d15bb32c99320054d153490ebe26888efe9812e3e4162b31b5eeb3" exitCode=0
Oct 31 02:18:22.612590 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.612117    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"
Oct 31 02:18:22.612590 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.612126    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" event={"ID":"176ca180-029d-4252-9ce9-a73297ffb286","Type":"ContainerDied","Data":"1fe59413b6d15bb32c99320054d153490ebe26888efe9812e3e4162b31b5eeb3"}
Oct 31 02:18:22.612590 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.612157    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5" event={"ID":"176ca180-029d-4252-9ce9-a73297ffb286","Type":"ContainerDied","Data":"e6b24f8c08d2e7d66229c687a28bd5aa0e3f225d6855f222737d08018c08f99a"}
Oct 31 02:18:22.614762 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.614737    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2" event={"ID":"8ac2ebbd-44b7-499a-be04-4d419002e32f","Type":"ContainerDied","Data":"cf470344a400f5e0edd3ca002700cd28eef350d0405b3209b8db36afb7cea36a"}
Oct 31 02:18:22.614905 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.614782    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2"
Oct 31 02:18:22.662036 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.662012    5650 scope.go:122] "RemoveContainer" containerID="ef5b8aad127c1dff1cc7462a1904495fc686f2e64d18ba9297fd686da26758ff"
Oct 31 02:18:22.662482 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:18:22.662433    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef5b8aad127c1dff1cc7462a1904495fc686f2e64d18ba9297fd686da26758ff\": container with ID starting with ef5b8aad127c1dff1cc7462a1904495fc686f2e64d18ba9297fd686da26758ff not found: ID does not exist" containerID="ef5b8aad127c1dff1cc7462a1904495fc686f2e64d18ba9297fd686da26758ff"
Oct 31 02:18:22.662580 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.662497    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef5b8aad127c1dff1cc7462a1904495fc686f2e64d18ba9297fd686da26758ff"} err="failed to get container status \"ef5b8aad127c1dff1cc7462a1904495fc686f2e64d18ba9297fd686da26758ff\": rpc error: code = NotFound desc = could not find container \"ef5b8aad127c1dff1cc7462a1904495fc686f2e64d18ba9297fd686da26758ff\": container with ID starting with ef5b8aad127c1dff1cc7462a1904495fc686f2e64d18ba9297fd686da26758ff not found: ID does not exist"
Oct 31 02:18:22.662580 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.662517    5650 scope.go:122] "RemoveContainer" containerID="1fe59413b6d15bb32c99320054d153490ebe26888efe9812e3e4162b31b5eeb3"
Oct 31 02:18:22.664177 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.664157    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr"]
Oct 31 02:18:22.667202 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.667178    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7a850e-6d8b6bd575-mwxgr"]
Oct 31 02:18:22.673269 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.673246    5650 scope.go:122] "RemoveContainer" containerID="1fe59413b6d15bb32c99320054d153490ebe26888efe9812e3e4162b31b5eeb3"
Oct 31 02:18:22.673628 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:18:22.673606    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fe59413b6d15bb32c99320054d153490ebe26888efe9812e3e4162b31b5eeb3\": container with ID starting with 1fe59413b6d15bb32c99320054d153490ebe26888efe9812e3e4162b31b5eeb3 not found: ID does not exist" containerID="1fe59413b6d15bb32c99320054d153490ebe26888efe9812e3e4162b31b5eeb3"
Oct 31 02:18:22.673696 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.673636    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fe59413b6d15bb32c99320054d153490ebe26888efe9812e3e4162b31b5eeb3"} err="failed to get container status \"1fe59413b6d15bb32c99320054d153490ebe26888efe9812e3e4162b31b5eeb3\": rpc error: code = NotFound desc = could not find container \"1fe59413b6d15bb32c99320054d153490ebe26888efe9812e3e4162b31b5eeb3\": container with ID starting with 1fe59413b6d15bb32c99320054d153490ebe26888efe9812e3e4162b31b5eeb3 not found: ID does not exist"
Oct 31 02:18:22.673696 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.673656    5650 scope.go:122] "RemoveContainer" containerID="1274a09d74b8206892ef582528e6f6de9b37ce25c728938292c30775e2aa1ef3"
Oct 31 02:18:22.679234 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.679210    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2"]
Oct 31 02:18:22.684740 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.684718    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7a850e-7947cfdfcf-8w2q2"]
Oct 31 02:18:22.698023 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.697993    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"]
Oct 31 02:18:22.704818 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:22.704789    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-0e8b5c/simulator-w1-0e8b5c-kserve-89647c967-7nbb5"]
Oct 31 02:18:24.122725 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:24.122678    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="176ca180-029d-4252-9ce9-a73297ffb286" path="/var/lib/kubelet/pods/176ca180-029d-4252-9ce9-a73297ffb286/volumes"
Oct 31 02:18:24.123433 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:24.123406    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d3621cc-41bb-4583-b189-ce7402a285cd" path="/var/lib/kubelet/pods/3d3621cc-41bb-4583-b189-ce7402a285cd/volumes"
Oct 31 02:18:24.124088 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:24.124063    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ac2ebbd-44b7-499a-be04-4d419002e32f" path="/var/lib/kubelet/pods/8ac2ebbd-44b7-499a-be04-4d419002e32f/volumes"
Oct 31 02:18:25.832106 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:25.832050    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"]
Oct 31 02:18:25.832926 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:25.832892    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf" podUID="f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" containerName="istio-proxy" containerID="cri-o://5a429146af346981ba9fcf8cc978728f4cddc165d49836dc3cba6c0131a4ae8e" gracePeriod=30
Oct 31 02:18:26.123921 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.123839    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq"]
Oct 31 02:18:26.124106 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.124078    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq" podUID="0731e478-fead-48bf-9de2-4cf3ee6f184c" containerName="maas-api" containerID="cri-o://b9006d3caeb1330191880231006d93a49bbd94c5e7329a01cfac2aafaaf2e190" gracePeriod=30
Oct 31 02:18:26.394761 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.394730    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq"
Oct 31 02:18:26.536554 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.536514    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0731e478-fead-48bf-9de2-4cf3ee6f184c-kube-api-access-lkgmt\" (UniqueName: \"kubernetes.io/projected/0731e478-fead-48bf-9de2-4cf3ee6f184c-kube-api-access-lkgmt\") pod \"0731e478-fead-48bf-9de2-4cf3ee6f184c\" (UID: \"0731e478-fead-48bf-9de2-4cf3ee6f184c\") "
Oct 31 02:18:26.536760 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.536591    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0731e478-fead-48bf-9de2-4cf3ee6f184c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0731e478-fead-48bf-9de2-4cf3ee6f184c-metrics-certs\") pod \"0731e478-fead-48bf-9de2-4cf3ee6f184c\" (UID: \"0731e478-fead-48bf-9de2-4cf3ee6f184c\") "
Oct 31 02:18:26.536760 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.536692    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0731e478-fead-48bf-9de2-4cf3ee6f184c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0731e478-fead-48bf-9de2-4cf3ee6f184c-maas-api-tls\") pod \"0731e478-fead-48bf-9de2-4cf3ee6f184c\" (UID: \"0731e478-fead-48bf-9de2-4cf3ee6f184c\") "
Oct 31 02:18:26.536760 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.536735    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0731e478-fead-48bf-9de2-4cf3ee6f184c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0731e478-fead-48bf-9de2-4cf3ee6f184c-openshift-service-ca\") pod \"0731e478-fead-48bf-9de2-4cf3ee6f184c\" (UID: \"0731e478-fead-48bf-9de2-4cf3ee6f184c\") "
Oct 31 02:18:26.537199 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.537154    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0731e478-fead-48bf-9de2-4cf3ee6f184c-openshift-service-ca" pod "0731e478-fead-48bf-9de2-4cf3ee6f184c" (UID: "0731e478-fead-48bf-9de2-4cf3ee6f184c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:26.537369 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.537289    5650 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0731e478-fead-48bf-9de2-4cf3ee6f184c-openshift-service-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:26.538993 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.538965    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0731e478-fead-48bf-9de2-4cf3ee6f184c-kube-api-access-lkgmt" pod "0731e478-fead-48bf-9de2-4cf3ee6f184c" (UID: "0731e478-fead-48bf-9de2-4cf3ee6f184c"). InnerVolumeSpecName "kube-api-access-lkgmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:26.539154 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.539135    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0731e478-fead-48bf-9de2-4cf3ee6f184c-metrics-certs" pod "0731e478-fead-48bf-9de2-4cf3ee6f184c" (UID: "0731e478-fead-48bf-9de2-4cf3ee6f184c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:18:26.539213 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.539153    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0731e478-fead-48bf-9de2-4cf3ee6f184c-maas-api-tls" pod "0731e478-fead-48bf-9de2-4cf3ee6f184c" (UID: "0731e478-fead-48bf-9de2-4cf3ee6f184c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:18:26.639062 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.638912    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkgmt\" (UniqueName: \"kubernetes.io/projected/0731e478-fead-48bf-9de2-4cf3ee6f184c-kube-api-access-lkgmt\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:26.639062 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.638952    5650 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0731e478-fead-48bf-9de2-4cf3ee6f184c-metrics-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:26.639062 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.638966    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0731e478-fead-48bf-9de2-4cf3ee6f184c-maas-api-tls\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:26.642673 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.642643    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf"]
Oct 31 02:18:26.643182 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.643149    5650 cpu_manager.go:414] "containerMap: removing container" podUID="0731e478-fead-48bf-9de2-4cf3ee6f184c" containerName="maas-api"
Oct 31 02:18:26.643182 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.643171    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0731e478-fead-48bf-9de2-4cf3ee6f184c" containerName="maas-api"
Oct 31 02:18:26.643427 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.643193    5650 cpu_manager.go:414] "containerMap: removing container" podUID="8ac2ebbd-44b7-499a-be04-4d419002e32f" containerName="payload-processing"
Oct 31 02:18:26.643427 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.643204    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac2ebbd-44b7-499a-be04-4d419002e32f" containerName="payload-processing"
Oct 31 02:18:26.643427 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.643217    5650 cpu_manager.go:414] "containerMap: removing container" podUID="1013a448-811b-4c57-ad4a-acb5d91553e4" containerName="revoke-keys"
Oct 31 02:18:26.643427 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.643223    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1013a448-811b-4c57-ad4a-acb5d91553e4" containerName="revoke-keys"
Oct 31 02:18:26.643427 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.643245    5650 cpu_manager.go:414] "containerMap: removing container" podUID="3d3621cc-41bb-4583-b189-ce7402a285cd" containerName="payload-pre-processing"
Oct 31 02:18:26.643427 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.643251    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3621cc-41bb-4583-b189-ce7402a285cd" containerName="payload-pre-processing"
Oct 31 02:18:26.643427 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.643267    5650 cpu_manager.go:414] "containerMap: removing container" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main"
Oct 31 02:18:26.643427 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.643275    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main"
Oct 31 02:18:26.643427 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.643288    5650 cpu_manager.go:414] "containerMap: removing container" podUID="c060bf51-21db-4f5b-889d-e655338a9bfa" containerName="istio-proxy"
Oct 31 02:18:26.643427 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.643293    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c060bf51-21db-4f5b-889d-e655338a9bfa" containerName="istio-proxy"
Oct 31 02:18:26.643427 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.643368    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="4238ae64-8bce-4c8d-8f14-339706f25bed" containerName="istio-proxy"
Oct 31 02:18:26.643427 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.643379    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="86d5dc42-ef1e-4ce5-9f29-386030d4524b" containerName="istio-proxy"
Oct 31 02:18:26.643427 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.643387    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d3621cc-41bb-4583-b189-ce7402a285cd" containerName="payload-pre-processing"
Oct 31 02:18:26.643427 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.643394    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="176ca180-029d-4252-9ce9-a73297ffb286" containerName="main"
Oct 31 02:18:26.643427 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.643404    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="0731e478-fead-48bf-9de2-4cf3ee6f184c" containerName="maas-api"
Oct 31 02:18:26.643427 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.643410    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac2ebbd-44b7-499a-be04-4d419002e32f" containerName="payload-processing"
Oct 31 02:18:26.643427 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.643417    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="1013a448-811b-4c57-ad4a-acb5d91553e4" containerName="revoke-keys"
Oct 31 02:18:26.643427 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.643424    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="c060bf51-21db-4f5b-889d-e655338a9bfa" containerName="istio-proxy"
Oct 31 02:18:26.646801 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.646776    5650 generic.go:356] "Generic (PLEG): container finished" podID="0731e478-fead-48bf-9de2-4cf3ee6f184c" containerID="b9006d3caeb1330191880231006d93a49bbd94c5e7329a01cfac2aafaaf2e190" exitCode=0
Oct 31 02:18:26.646801 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.646794    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq" event={"ID":"0731e478-fead-48bf-9de2-4cf3ee6f184c","Type":"ContainerDied","Data":"b9006d3caeb1330191880231006d93a49bbd94c5e7329a01cfac2aafaaf2e190"}
Oct 31 02:18:26.647011 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.646826    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq" event={"ID":"0731e478-fead-48bf-9de2-4cf3ee6f184c","Type":"ContainerDied","Data":"a40e0f92b0a7de1f74c6a1b0c34b6ce1c239763168dedd724c568a170ac61bfe"}
Oct 31 02:18:26.647011 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.646849    5650 scope.go:122] "RemoveContainer" containerID="b9006d3caeb1330191880231006d93a49bbd94c5e7329a01cfac2aafaaf2e190"
Oct 31 02:18:26.647011 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.646894    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq"
Oct 31 02:18:26.647011 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.646902    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf"
Oct 31 02:18:26.658379 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.658350    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf"]
Oct 31 02:18:26.659801 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.659780    5650 scope.go:122] "RemoveContainer" containerID="b9006d3caeb1330191880231006d93a49bbd94c5e7329a01cfac2aafaaf2e190"
Oct 31 02:18:26.660200 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:18:26.660174    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9006d3caeb1330191880231006d93a49bbd94c5e7329a01cfac2aafaaf2e190\": container with ID starting with b9006d3caeb1330191880231006d93a49bbd94c5e7329a01cfac2aafaaf2e190 not found: ID does not exist" containerID="b9006d3caeb1330191880231006d93a49bbd94c5e7329a01cfac2aafaaf2e190"
Oct 31 02:18:26.660269 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.660214    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9006d3caeb1330191880231006d93a49bbd94c5e7329a01cfac2aafaaf2e190"} err="failed to get container status \"b9006d3caeb1330191880231006d93a49bbd94c5e7329a01cfac2aafaaf2e190\": rpc error: code = NotFound desc = could not find container \"b9006d3caeb1330191880231006d93a49bbd94c5e7329a01cfac2aafaaf2e190\": container with ID starting with b9006d3caeb1330191880231006d93a49bbd94c5e7329a01cfac2aafaaf2e190 not found: ID does not exist"
Oct 31 02:18:26.663998 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.661421    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:18:26.663998 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.661476    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:18:26.702664 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.702634    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq"]
Oct 31 02:18:26.706785 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.706752    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-0e8b5c-5d5d6cbbb9-5crmq"]
Oct 31 02:18:26.740615 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.740575    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d56aacf5-6679-469c-ba41-c68d489ef628-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf\" (UID: \"d56aacf5-6679-469c-ba41-c68d489ef628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf"
Oct 31 02:18:26.740784 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.740623    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skt6s\" (UniqueName: \"kubernetes.io/projected/d56aacf5-6679-469c-ba41-c68d489ef628-kube-api-access-skt6s\") pod \"maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf\" (UID: \"d56aacf5-6679-469c-ba41-c68d489ef628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf"
Oct 31 02:18:26.740784 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.740712    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d56aacf5-6679-469c-ba41-c68d489ef628-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf\" (UID: \"d56aacf5-6679-469c-ba41-c68d489ef628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf"
Oct 31 02:18:26.841899 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.841854    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d56aacf5-6679-469c-ba41-c68d489ef628-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf\" (UID: \"d56aacf5-6679-469c-ba41-c68d489ef628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf"
Oct 31 02:18:26.842287 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.842021    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d56aacf5-6679-469c-ba41-c68d489ef628-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf\" (UID: \"d56aacf5-6679-469c-ba41-c68d489ef628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf"
Oct 31 02:18:26.842287 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.842045    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skt6s\" (UniqueName: \"kubernetes.io/projected/d56aacf5-6679-469c-ba41-c68d489ef628-kube-api-access-skt6s\") pod \"maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf\" (UID: \"d56aacf5-6679-469c-ba41-c68d489ef628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf"
Oct 31 02:18:26.842608 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.842572    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d56aacf5-6679-469c-ba41-c68d489ef628-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf\" (UID: \"d56aacf5-6679-469c-ba41-c68d489ef628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf"
Oct 31 02:18:26.844407 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.844385    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d56aacf5-6679-469c-ba41-c68d489ef628-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf\" (UID: \"d56aacf5-6679-469c-ba41-c68d489ef628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf"
Oct 31 02:18:26.855306 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.855281    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skt6s\" (UniqueName: \"kubernetes.io/projected/d56aacf5-6679-469c-ba41-c68d489ef628-kube-api-access-skt6s\") pod \"maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf\" (UID: \"d56aacf5-6679-469c-ba41-c68d489ef628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf"
Oct 31 02:18:26.969979 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:26.969935    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf"
Oct 31 02:18:27.253049 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:27.253013    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf"]
Oct 31 02:18:27.254033 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:18:27.254003    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56aacf5_6679_469c_ba41_c68d489ef628.slice/crio-cc82aeba1b780ab723286969e0d4e19bc459377f1618e72cf2c0462551420eaf WatchSource:0}: Error finding container cc82aeba1b780ab723286969e0d4e19bc459377f1618e72cf2c0462551420eaf: Status 404 returned error can't find the container with id cc82aeba1b780ab723286969e0d4e19bc459377f1618e72cf2c0462551420eaf
Oct 31 02:18:27.656691 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:27.656653    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf" event={"ID":"d56aacf5-6679-469c-ba41-c68d489ef628","Type":"ContainerStarted","Data":"674f72f77783672fc96b233ebeab05186de9c4f20f490a2e3bcf1eed456b2e66"}
Oct 31 02:18:27.656890 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:27.656713    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf" event={"ID":"d56aacf5-6679-469c-ba41-c68d489ef628","Type":"ContainerStarted","Data":"cc82aeba1b780ab723286969e0d4e19bc459377f1618e72cf2c0462551420eaf"}
Oct 31 02:18:27.679554 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:27.679496    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf" podStartSLOduration=1.679467096 podStartE2EDuration="1.679467096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:18:27.677277519 +0000 UTC m=+3463.680991815" watchObservedRunningTime="2026-10-31 02:18:27.679467096 +0000 UTC m=+3463.683181378"
Oct 31 02:18:27.740166 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:27.740120    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l" podUID="8969a12d-73b4-4d52-a144-918c72ded4bc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:18:28.118298 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:28.118268    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0731e478-fead-48bf-9de2-4cf3ee6f184c" path="/var/lib/kubelet/pods/0731e478-fead-48bf-9de2-4cf3ee6f184c/volumes"
Oct 31 02:18:28.558167 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:18:28.558127    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:18:28.665357 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:28.665325    5650 generic.go:356] "Generic (PLEG): container finished" podID="d56aacf5-6679-469c-ba41-c68d489ef628" containerID="674f72f77783672fc96b233ebeab05186de9c4f20f490a2e3bcf1eed456b2e66" exitCode=0
Oct 31 02:18:28.665627 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:28.665414    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf" event={"ID":"d56aacf5-6679-469c-ba41-c68d489ef628","Type":"ContainerDied","Data":"674f72f77783672fc96b233ebeab05186de9c4f20f490a2e3bcf1eed456b2e66"}
Oct 31 02:18:29.823739 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:29.823711    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf"
Oct 31 02:18:29.981015 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:29.980980    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d56aacf5-6679-469c-ba41-c68d489ef628-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d56aacf5-6679-469c-ba41-c68d489ef628-maas-api-xks-ca\") pod \"d56aacf5-6679-469c-ba41-c68d489ef628\" (UID: \"d56aacf5-6679-469c-ba41-c68d489ef628\") "
Oct 31 02:18:29.981219 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:29.981152    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d56aacf5-6679-469c-ba41-c68d489ef628-kube-api-access-skt6s\" (UniqueName: \"kubernetes.io/projected/d56aacf5-6679-469c-ba41-c68d489ef628-kube-api-access-skt6s\") pod \"d56aacf5-6679-469c-ba41-c68d489ef628\" (UID: \"d56aacf5-6679-469c-ba41-c68d489ef628\") "
Oct 31 02:18:29.981219 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:29.981216    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d56aacf5-6679-469c-ba41-c68d489ef628-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d56aacf5-6679-469c-ba41-c68d489ef628-maas-api-openshift-ca\") pod \"d56aacf5-6679-469c-ba41-c68d489ef628\" (UID: \"d56aacf5-6679-469c-ba41-c68d489ef628\") "
Oct 31 02:18:29.981611 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:29.981581    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d56aacf5-6679-469c-ba41-c68d489ef628-maas-api-openshift-ca" pod "d56aacf5-6679-469c-ba41-c68d489ef628" (UID: "d56aacf5-6679-469c-ba41-c68d489ef628"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:29.981709 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:29.981692    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d56aacf5-6679-469c-ba41-c68d489ef628-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:29.983220 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:29.983197    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d56aacf5-6679-469c-ba41-c68d489ef628-maas-api-xks-ca" pod "d56aacf5-6679-469c-ba41-c68d489ef628" (UID: "d56aacf5-6679-469c-ba41-c68d489ef628"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:18:29.983459 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:29.983430    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d56aacf5-6679-469c-ba41-c68d489ef628-kube-api-access-skt6s" pod "d56aacf5-6679-469c-ba41-c68d489ef628" (UID: "d56aacf5-6679-469c-ba41-c68d489ef628"). InnerVolumeSpecName "kube-api-access-skt6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:30.083062 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:30.083026    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d56aacf5-6679-469c-ba41-c68d489ef628-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:30.083062 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:30.083057    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skt6s\" (UniqueName: \"kubernetes.io/projected/d56aacf5-6679-469c-ba41-c68d489ef628-kube-api-access-skt6s\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:30.679766 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:30.679729    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf"
Oct 31 02:18:30.679926 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:30.679769    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf" event={"ID":"d56aacf5-6679-469c-ba41-c68d489ef628","Type":"ContainerDied","Data":"cc82aeba1b780ab723286969e0d4e19bc459377f1618e72cf2c0462551420eaf"}
Oct 31 02:18:30.679926 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:30.679823    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc82aeba1b780ab723286969e0d4e19bc459377f1618e72cf2c0462551420eaf"
Oct 31 02:18:31.107101 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.107075    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 02:18:31.193908 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.193865    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-data\") pod \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") "
Oct 31 02:18:31.194110 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.193930    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-workload-socket\") pod \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") "
Oct 31 02:18:31.194110 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.193987    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-envoy\") pod \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") "
Oct 31 02:18:31.194110 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.194017    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-podinfo\") pod \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") "
Oct 31 02:18:31.194110 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.194046    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istiod-ca-cert\") pod \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") "
Oct 31 02:18:31.194110 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.194093    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-token\" (UniqueName: \"kubernetes.io/projected/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-token\") pod \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") "
Oct 31 02:18:31.194407 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.194150    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-workload-certs\") pod \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") "
Oct 31 02:18:31.194407 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.194184    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-credential-socket\") pod \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") "
Oct 31 02:18:31.194407 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.194207    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-data" pod "f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" (UID: "f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:31.194407 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.194251    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-kube-api-access-rcdv9\" (UniqueName: \"kubernetes.io/projected/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-kube-api-access-rcdv9\") pod \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\" (UID: \"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db\") "
Oct 31 02:18:31.194407 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.194397    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-workload-socket" pod "f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" (UID: "f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:31.194816 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.194736    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istiod-ca-cert" pod "f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" (UID: "f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:31.194816 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.194770    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-credential-socket" pod "f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" (UID: "f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:31.195021 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.194923    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:31.195021 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.194944    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:31.195021 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.194961    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:31.195021 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.194975    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:31.195366 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.195286    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-workload-certs" pod "f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" (UID: "f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:31.196434 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.196409    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-envoy" pod "f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" (UID: "f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:31.196746 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.196726    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-kube-api-access-rcdv9" pod "f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" (UID: "f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db"). InnerVolumeSpecName "kube-api-access-rcdv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:31.196878 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.196862    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-token" pod "f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" (UID: "f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:31.197169 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.197138    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-podinfo" pod "f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" (UID: "f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:18:31.296584 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.296545    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:31.296584 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.296586    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:31.296808 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.296603    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:31.296808 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.296619    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:31.296808 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.296629    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcdv9\" (UniqueName: \"kubernetes.io/projected/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db-kube-api-access-rcdv9\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:31.367265 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.362735    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"]
Oct 31 02:18:31.688709 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.688620    5650 generic.go:356] "Generic (PLEG): container finished" podID="f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" containerID="5a429146af346981ba9fcf8cc978728f4cddc165d49836dc3cba6c0131a4ae8e" exitCode=0
Oct 31 02:18:31.688709 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.688679    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"
Oct 31 02:18:31.688709 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.688688    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf" event={"ID":"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db","Type":"ContainerDied","Data":"5a429146af346981ba9fcf8cc978728f4cddc165d49836dc3cba6c0131a4ae8e"}
Oct 31 02:18:31.688964 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.688773    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf" event={"ID":"f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db","Type":"ContainerDied","Data":"67066061c7e9ac2e4022e2ed8ea9aaf658ed691d69d785d3810325dec68c9a69"}
Oct 31 02:18:31.688964 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.688802    5650 scope.go:122] "RemoveContainer" containerID="5a429146af346981ba9fcf8cc978728f4cddc165d49836dc3cba6c0131a4ae8e"
Oct 31 02:18:31.704848 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.704828    5650 scope.go:122] "RemoveContainer" containerID="a40a5f0f3953f38da44794fc7ab5fce6c1efea4f791b8205452a6b6d054beb6d"
Oct 31 02:18:31.715714 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.715626    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"]
Oct 31 02:18:31.715787 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.715764    5650 scope.go:122] "RemoveContainer" containerID="5a429146af346981ba9fcf8cc978728f4cddc165d49836dc3cba6c0131a4ae8e"
Oct 31 02:18:31.716097 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:18:31.716065    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a429146af346981ba9fcf8cc978728f4cddc165d49836dc3cba6c0131a4ae8e\": container with ID starting with 5a429146af346981ba9fcf8cc978728f4cddc165d49836dc3cba6c0131a4ae8e not found: ID does not exist" containerID="5a429146af346981ba9fcf8cc978728f4cddc165d49836dc3cba6c0131a4ae8e"
Oct 31 02:18:31.716226 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.716103    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a429146af346981ba9fcf8cc978728f4cddc165d49836dc3cba6c0131a4ae8e"} err="failed to get container status \"5a429146af346981ba9fcf8cc978728f4cddc165d49836dc3cba6c0131a4ae8e\": rpc error: code = NotFound desc = could not find container \"5a429146af346981ba9fcf8cc978728f4cddc165d49836dc3cba6c0131a4ae8e\": container with ID starting with 5a429146af346981ba9fcf8cc978728f4cddc165d49836dc3cba6c0131a4ae8e not found: ID does not exist"
Oct 31 02:18:31.716226 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.716127    5650 scope.go:122] "RemoveContainer" containerID="a40a5f0f3953f38da44794fc7ab5fce6c1efea4f791b8205452a6b6d054beb6d"
Oct 31 02:18:31.716764 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:18:31.716695    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a40a5f0f3953f38da44794fc7ab5fce6c1efea4f791b8205452a6b6d054beb6d\": container with ID starting with a40a5f0f3953f38da44794fc7ab5fce6c1efea4f791b8205452a6b6d054beb6d not found: ID does not exist" containerID="a40a5f0f3953f38da44794fc7ab5fce6c1efea4f791b8205452a6b6d054beb6d"
Oct 31 02:18:31.716764 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.716733    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a40a5f0f3953f38da44794fc7ab5fce6c1efea4f791b8205452a6b6d054beb6d"} err="failed to get container status \"a40a5f0f3953f38da44794fc7ab5fce6c1efea4f791b8205452a6b6d054beb6d\": rpc error: code = NotFound desc = could not find container \"a40a5f0f3953f38da44794fc7ab5fce6c1efea4f791b8205452a6b6d054beb6d\": container with ID starting with a40a5f0f3953f38da44794fc7ab5fce6c1efea4f791b8205452a6b6d054beb6d not found: ID does not exist"
Oct 31 02:18:31.720501 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:31.720476    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-gwqhf"]
Oct 31 02:18:32.118551 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:32.118519    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" path="/var/lib/kubelet/pods/f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db/volumes"
Oct 31 02:18:33.813704 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:33.813656    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" containerID="cri-o://951f29eef385521b041d51dc2ef435ac0dfe4d1f68768accc0e23a317a0d052b" gracePeriod=45
Oct 31 02:18:33.817808 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:33.817768    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main" containerID="cri-o://bc6e7782cd99eccb1f87e8dae71dfee672433454fef75c490095dfcc5ba346d9" gracePeriod=45
Oct 31 02:18:33.835925 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:33.835886    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" containerID="cri-o://be9a7c7decc2311a3c5ce38a880ecc4dcab782986a1adc40ef55fe23d778b931" gracePeriod=45
Oct 31 02:18:33.971829 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:33.971698    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": dial tcp 10.134.0.76:8000: connect: connection refused"
Oct 31 02:18:33.999172 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:33.999130    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": dial tcp 10.134.0.62:8000: connect: connection refused"
Oct 31 02:18:34.106188 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.106154    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 02:18:34.149814 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.149783    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 02:18:34.165370 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.165342    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 02:18:34.241355 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.241314    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/210cc1c7-6a02-495c-89f1-6dc889f8bf11-tls-certs\" (UniqueName: \"kubernetes.io/secret/210cc1c7-6a02-495c-89f1-6dc889f8bf11-tls-certs\") pod \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\" (UID: \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\") "
Oct 31 02:18:34.241530 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.241412    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-tmp-dir\") pod \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\" (UID: \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\") "
Oct 31 02:18:34.241579 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.241527    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cbae488-58ba-454e-874a-8849b71ec03a-kube-api-access-lqmlr\" (UniqueName: \"kubernetes.io/projected/0cbae488-58ba-454e-874a-8849b71ec03a-kube-api-access-lqmlr\") pod \"0cbae488-58ba-454e-874a-8849b71ec03a\" (UID: \"0cbae488-58ba-454e-874a-8849b71ec03a\") "
Oct 31 02:18:34.241579 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.241566    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-home\" (UniqueName: \"kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-home\") pod \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\" (UID: \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\") "
Oct 31 02:18:34.241714 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.241601    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-home\" (UniqueName: \"kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-home\") pod \"0cbae488-58ba-454e-874a-8849b71ec03a\" (UID: \"0cbae488-58ba-454e-874a-8849b71ec03a\") "
Oct 31 02:18:34.241714 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.241677    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-dshm\") pod \"0cbae488-58ba-454e-874a-8849b71ec03a\" (UID: \"0cbae488-58ba-454e-874a-8849b71ec03a\") "
Oct 31 02:18:34.241821 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.241711    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cbae488-58ba-454e-874a-8849b71ec03a-tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbae488-58ba-454e-874a-8849b71ec03a-tls-certs\") pod \"0cbae488-58ba-454e-874a-8849b71ec03a\" (UID: \"0cbae488-58ba-454e-874a-8849b71ec03a\") "
Oct 31 02:18:34.241821 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.241745    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-dshm\" (UniqueName: \"kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-dshm\") pod \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\" (UID: \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\") "
Oct 31 02:18:34.241821 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.241777    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-model-cache\") pod \"0cbae488-58ba-454e-874a-8849b71ec03a\" (UID: \"0cbae488-58ba-454e-874a-8849b71ec03a\") "
Oct 31 02:18:34.241821 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.241787    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-tmp-dir" pod "210cc1c7-6a02-495c-89f1-6dc889f8bf11" (UID: "210cc1c7-6a02-495c-89f1-6dc889f8bf11"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:34.242058 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.241821    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/210cc1c7-6a02-495c-89f1-6dc889f8bf11-kube-api-access-42dwd\" (UniqueName: \"kubernetes.io/projected/210cc1c7-6a02-495c-89f1-6dc889f8bf11-kube-api-access-42dwd\") pod \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\" (UID: \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\") "
Oct 31 02:18:34.242058 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.241844    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-home" pod "210cc1c7-6a02-495c-89f1-6dc889f8bf11" (UID: "210cc1c7-6a02-495c-89f1-6dc889f8bf11"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:34.242058 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.241889    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-model-cache\") pod \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\" (UID: \"210cc1c7-6a02-495c-89f1-6dc889f8bf11\") "
Oct 31 02:18:34.242058 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.241982    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-tmp-dir\") pod \"0cbae488-58ba-454e-874a-8849b71ec03a\" (UID: \"0cbae488-58ba-454e-874a-8849b71ec03a\") "
Oct 31 02:18:34.242550 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.242527    5650 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-tmp-dir\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:34.242550 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.242549    5650 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-home\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:34.242711 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.242593    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-home" pod "0cbae488-58ba-454e-874a-8849b71ec03a" (UID: "0cbae488-58ba-454e-874a-8849b71ec03a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:34.242972 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.242945    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-model-cache" pod "210cc1c7-6a02-495c-89f1-6dc889f8bf11" (UID: "210cc1c7-6a02-495c-89f1-6dc889f8bf11"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:34.243311 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.243259    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-model-cache" pod "0cbae488-58ba-454e-874a-8849b71ec03a" (UID: "0cbae488-58ba-454e-874a-8849b71ec03a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:34.243539 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.243515    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-tmp-dir" pod "0cbae488-58ba-454e-874a-8849b71ec03a" (UID: "0cbae488-58ba-454e-874a-8849b71ec03a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:34.244538 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.244511    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/210cc1c7-6a02-495c-89f1-6dc889f8bf11-tls-certs" pod "210cc1c7-6a02-495c-89f1-6dc889f8bf11" (UID: "210cc1c7-6a02-495c-89f1-6dc889f8bf11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:18:34.245360 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.245331    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-dshm" pod "210cc1c7-6a02-495c-89f1-6dc889f8bf11" (UID: "210cc1c7-6a02-495c-89f1-6dc889f8bf11"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:34.245484 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.245411    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cbae488-58ba-454e-874a-8849b71ec03a-tls-certs" pod "0cbae488-58ba-454e-874a-8849b71ec03a" (UID: "0cbae488-58ba-454e-874a-8849b71ec03a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:18:34.245804 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.245774    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbae488-58ba-454e-874a-8849b71ec03a-kube-api-access-lqmlr" pod "0cbae488-58ba-454e-874a-8849b71ec03a" (UID: "0cbae488-58ba-454e-874a-8849b71ec03a"). InnerVolumeSpecName "kube-api-access-lqmlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:34.246435 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.246415    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-dshm" pod "0cbae488-58ba-454e-874a-8849b71ec03a" (UID: "0cbae488-58ba-454e-874a-8849b71ec03a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:34.246753 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.246731    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210cc1c7-6a02-495c-89f1-6dc889f8bf11-kube-api-access-42dwd" pod "210cc1c7-6a02-495c-89f1-6dc889f8bf11" (UID: "210cc1c7-6a02-495c-89f1-6dc889f8bf11"). InnerVolumeSpecName "kube-api-access-42dwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:34.343366 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.343276    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-tls-certs\" (UniqueName: \"kubernetes.io/secret/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-tls-certs\") pod \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\" (UID: \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\") "
Oct 31 02:18:34.343551 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.343480    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-kube-api-access-nscds\" (UniqueName: \"kubernetes.io/projected/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-kube-api-access-nscds\") pod \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\" (UID: \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\") "
Oct 31 02:18:34.343551 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.343525    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-model-cache\") pod \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\" (UID: \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\") "
Oct 31 02:18:34.343697 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.343567    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-tmp-dir\") pod \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\" (UID: \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\") "
Oct 31 02:18:34.343697 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.343587    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-dshm\") pod \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\" (UID: \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\") "
Oct 31 02:18:34.343697 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.343668    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-home\" (UniqueName: \"kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-home\") pod \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\" (UID: \"0cde8fd2-23bc-46c4-8f56-f6e17339c50d\") "
Oct 31 02:18:34.343973 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.343929    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-tmp-dir" pod "0cde8fd2-23bc-46c4-8f56-f6e17339c50d" (UID: "0cde8fd2-23bc-46c4-8f56-f6e17339c50d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:34.343973 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.343944    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-model-cache" pod "0cde8fd2-23bc-46c4-8f56-f6e17339c50d" (UID: "0cde8fd2-23bc-46c4-8f56-f6e17339c50d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:34.344225 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.344128    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-home" pod "0cde8fd2-23bc-46c4-8f56-f6e17339c50d" (UID: "0cde8fd2-23bc-46c4-8f56-f6e17339c50d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:34.344225 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.344190    5650 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-model-cache\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:34.344225 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.344220    5650 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-tmp-dir\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:34.344361 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.344235    5650 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210cc1c7-6a02-495c-89f1-6dc889f8bf11-tls-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:34.344361 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.344249    5650 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-model-cache\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:34.344361 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.344262    5650 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-tmp-dir\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:34.344361 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.344276    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqmlr\" (UniqueName: \"kubernetes.io/projected/0cbae488-58ba-454e-874a-8849b71ec03a-kube-api-access-lqmlr\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:34.344361 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.344292    5650 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-home\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:34.344361 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.344306    5650 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-dshm\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:34.344361 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.344318    5650 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cbae488-58ba-454e-874a-8849b71ec03a-tls-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:34.344361 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.344331    5650 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210cc1c7-6a02-495c-89f1-6dc889f8bf11-dshm\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:34.344361 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.344344    5650 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cbae488-58ba-454e-874a-8849b71ec03a-model-cache\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:34.344361 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.344361    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42dwd\" (UniqueName: \"kubernetes.io/projected/210cc1c7-6a02-495c-89f1-6dc889f8bf11-kube-api-access-42dwd\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:34.345812 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.345779    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-kube-api-access-nscds" pod "0cde8fd2-23bc-46c4-8f56-f6e17339c50d" (UID: "0cde8fd2-23bc-46c4-8f56-f6e17339c50d"). InnerVolumeSpecName "kube-api-access-nscds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:34.345932 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.345878    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-tls-certs" pod "0cde8fd2-23bc-46c4-8f56-f6e17339c50d" (UID: "0cde8fd2-23bc-46c4-8f56-f6e17339c50d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:18:34.346025 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.345939    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-dshm" pod "0cde8fd2-23bc-46c4-8f56-f6e17339c50d" (UID: "0cde8fd2-23bc-46c4-8f56-f6e17339c50d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:34.445938 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.445898    5650 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-tls-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:34.445938 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.445935    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nscds\" (UniqueName: \"kubernetes.io/projected/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-kube-api-access-nscds\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:34.445938 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.445945    5650 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-dshm\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:34.446202 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.445954    5650 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cde8fd2-23bc-46c4-8f56-f6e17339c50d-home\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:34.713229 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.713194    5650 generic.go:356] "Generic (PLEG): container finished" podID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerID="be9a7c7decc2311a3c5ce38a880ecc4dcab782986a1adc40ef55fe23d778b931" exitCode=0
Oct 31 02:18:34.713435 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.713240    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"
Oct 31 02:18:34.713435 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.713283    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" event={"ID":"0cde8fd2-23bc-46c4-8f56-f6e17339c50d","Type":"ContainerDied","Data":"be9a7c7decc2311a3c5ce38a880ecc4dcab782986a1adc40ef55fe23d778b931"}
Oct 31 02:18:34.713435 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.713327    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9" event={"ID":"0cde8fd2-23bc-46c4-8f56-f6e17339c50d","Type":"ContainerDied","Data":"ba0612ecac6f18630cd148e4628193307b34484517de075c6762cc31d32a0158"}
Oct 31 02:18:34.713435 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.713347    5650 scope.go:122] "RemoveContainer" containerID="be9a7c7decc2311a3c5ce38a880ecc4dcab782986a1adc40ef55fe23d778b931"
Oct 31 02:18:34.714739 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.714716    5650 generic.go:356] "Generic (PLEG): container finished" podID="0cbae488-58ba-454e-874a-8849b71ec03a" containerID="951f29eef385521b041d51dc2ef435ac0dfe4d1f68768accc0e23a317a0d052b" exitCode=0
Oct 31 02:18:34.714919 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.714745    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" event={"ID":"0cbae488-58ba-454e-874a-8849b71ec03a","Type":"ContainerDied","Data":"951f29eef385521b041d51dc2ef435ac0dfe4d1f68768accc0e23a317a0d052b"}
Oct 31 02:18:34.714919 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.714764    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"
Oct 31 02:18:34.714919 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.714786    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47" event={"ID":"0cbae488-58ba-454e-874a-8849b71ec03a","Type":"ContainerDied","Data":"2e184427c35997bbc35cf2a69ca7e2093a420598ce24ac3bc2851cef7322eb53"}
Oct 31 02:18:34.716273 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.716247    5650 generic.go:356] "Generic (PLEG): container finished" podID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerID="bc6e7782cd99eccb1f87e8dae71dfee672433454fef75c490095dfcc5ba346d9" exitCode=0
Oct 31 02:18:34.716382 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.716336    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"
Oct 31 02:18:34.716458 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.716389    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" event={"ID":"210cc1c7-6a02-495c-89f1-6dc889f8bf11","Type":"ContainerDied","Data":"bc6e7782cd99eccb1f87e8dae71dfee672433454fef75c490095dfcc5ba346d9"}
Oct 31 02:18:34.716458 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.716419    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n" event={"ID":"210cc1c7-6a02-495c-89f1-6dc889f8bf11","Type":"ContainerDied","Data":"2f8c1ef11d4dcc1b19aed50a0e610b0afb50779e96df34c7bdecdc26c4586188"}
Oct 31 02:18:34.726877 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.726848    5650 scope.go:122] "RemoveContainer" containerID="be9a7c7decc2311a3c5ce38a880ecc4dcab782986a1adc40ef55fe23d778b931"
Oct 31 02:18:34.727716 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:18:34.727563    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9a7c7decc2311a3c5ce38a880ecc4dcab782986a1adc40ef55fe23d778b931\": container with ID starting with be9a7c7decc2311a3c5ce38a880ecc4dcab782986a1adc40ef55fe23d778b931 not found: ID does not exist" containerID="be9a7c7decc2311a3c5ce38a880ecc4dcab782986a1adc40ef55fe23d778b931"
Oct 31 02:18:34.727916 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.727726    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9a7c7decc2311a3c5ce38a880ecc4dcab782986a1adc40ef55fe23d778b931"} err="failed to get container status \"be9a7c7decc2311a3c5ce38a880ecc4dcab782986a1adc40ef55fe23d778b931\": rpc error: code = NotFound desc = could not find container \"be9a7c7decc2311a3c5ce38a880ecc4dcab782986a1adc40ef55fe23d778b931\": container with ID starting with be9a7c7decc2311a3c5ce38a880ecc4dcab782986a1adc40ef55fe23d778b931 not found: ID does not exist"
Oct 31 02:18:34.728011 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.727921    5650 scope.go:122] "RemoveContainer" containerID="951f29eef385521b041d51dc2ef435ac0dfe4d1f68768accc0e23a317a0d052b"
Oct 31 02:18:34.742577 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.742544    5650 scope.go:122] "RemoveContainer" containerID="951f29eef385521b041d51dc2ef435ac0dfe4d1f68768accc0e23a317a0d052b"
Oct 31 02:18:34.742958 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:18:34.742933    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"951f29eef385521b041d51dc2ef435ac0dfe4d1f68768accc0e23a317a0d052b\": container with ID starting with 951f29eef385521b041d51dc2ef435ac0dfe4d1f68768accc0e23a317a0d052b not found: ID does not exist" containerID="951f29eef385521b041d51dc2ef435ac0dfe4d1f68768accc0e23a317a0d052b"
Oct 31 02:18:34.743045 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.742966    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"951f29eef385521b041d51dc2ef435ac0dfe4d1f68768accc0e23a317a0d052b"} err="failed to get container status \"951f29eef385521b041d51dc2ef435ac0dfe4d1f68768accc0e23a317a0d052b\": rpc error: code = NotFound desc = could not find container \"951f29eef385521b041d51dc2ef435ac0dfe4d1f68768accc0e23a317a0d052b\": container with ID starting with 951f29eef385521b041d51dc2ef435ac0dfe4d1f68768accc0e23a317a0d052b not found: ID does not exist"
Oct 31 02:18:34.743045 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.742984    5650 scope.go:122] "RemoveContainer" containerID="bc6e7782cd99eccb1f87e8dae71dfee672433454fef75c490095dfcc5ba346d9"
Oct 31 02:18:34.745338 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.745314    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"]
Oct 31 02:18:34.749201 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.749178    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-6ae519/simulator-w2-6ae519-kserve-7785bbbc46-tnvd9"]
Oct 31 02:18:34.767606 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.767565    5650 scope.go:122] "RemoveContainer" containerID="bc6e7782cd99eccb1f87e8dae71dfee672433454fef75c490095dfcc5ba346d9"
Oct 31 02:18:34.768131 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:18:34.768094    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc6e7782cd99eccb1f87e8dae71dfee672433454fef75c490095dfcc5ba346d9\": container with ID starting with bc6e7782cd99eccb1f87e8dae71dfee672433454fef75c490095dfcc5ba346d9 not found: ID does not exist" containerID="bc6e7782cd99eccb1f87e8dae71dfee672433454fef75c490095dfcc5ba346d9"
Oct 31 02:18:34.768230 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.768145    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc6e7782cd99eccb1f87e8dae71dfee672433454fef75c490095dfcc5ba346d9"} err="failed to get container status \"bc6e7782cd99eccb1f87e8dae71dfee672433454fef75c490095dfcc5ba346d9\": rpc error: code = NotFound desc = could not find container \"bc6e7782cd99eccb1f87e8dae71dfee672433454fef75c490095dfcc5ba346d9\": container with ID starting with bc6e7782cd99eccb1f87e8dae71dfee672433454fef75c490095dfcc5ba346d9 not found: ID does not exist"
Oct 31 02:18:34.768491 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.768431    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"]
Oct 31 02:18:34.771417 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.771387    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-6ae519/distinct-w2-6ae519-kserve-5f68644bdb-6r98n"]
Oct 31 02:18:34.782398 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.782369    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"]
Oct 31 02:18:34.786455 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:34.786418    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-6ae519/distinct-2-w2-6ae519-kserve-69c7f49fd5-9gf47"]
Oct 31 02:18:35.018712 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:35.018615    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"]
Oct 31 02:18:35.020182 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:35.020148    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc" podUID="bb3603d0-f3c8-4bbc-96c7-366b59bffdce" containerName="istio-proxy" containerID="cri-o://1b460b3b1717dfb6801690ad4515a7f922a63a29cc4c0260e4f3bd715062a724" gracePeriod=30
Oct 31 02:18:36.118034 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:36.118000    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" path="/var/lib/kubelet/pods/0cbae488-58ba-454e-874a-8849b71ec03a/volumes"
Oct 31 02:18:36.118511 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:36.118435    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" path="/var/lib/kubelet/pods/0cde8fd2-23bc-46c4-8f56-f6e17339c50d/volumes"
Oct 31 02:18:36.118941 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:36.118928    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" path="/var/lib/kubelet/pods/210cc1c7-6a02-495c-89f1-6dc889f8bf11/volumes"
Oct 31 02:18:36.805842 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:36.805808    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn"]
Oct 31 02:18:37.120040 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.119944    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-359bbaa1-z4nvh"]
Oct 31 02:18:37.120545 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.120513    5650 cpu_manager.go:414] "containerMap: removing container" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main"
Oct 31 02:18:37.120545 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.120526    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main"
Oct 31 02:18:37.120545 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.120543    5650 cpu_manager.go:414] "containerMap: removing container" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main"
Oct 31 02:18:37.120716 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.120548    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main"
Oct 31 02:18:37.120716 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.120557    5650 cpu_manager.go:414] "containerMap: removing container" podUID="f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" containerName="istio-proxy"
Oct 31 02:18:37.120716 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.120564    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" containerName="istio-proxy"
Oct 31 02:18:37.120716 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.120582    5650 cpu_manager.go:414] "containerMap: removing container" podUID="d56aacf5-6679-469c-ba41-c68d489ef628" containerName="revoke-keys"
Oct 31 02:18:37.120716 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.120587    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d56aacf5-6679-469c-ba41-c68d489ef628" containerName="revoke-keys"
Oct 31 02:18:37.120716 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.120595    5650 cpu_manager.go:414] "containerMap: removing container" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main"
Oct 31 02:18:37.120716 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.120599    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main"
Oct 31 02:18:37.120716 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.120680    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" containerName="istio-proxy"
Oct 31 02:18:37.120716 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.120688    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="210cc1c7-6a02-495c-89f1-6dc889f8bf11" containerName="main"
Oct 31 02:18:37.120716 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.120697    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cbae488-58ba-454e-874a-8849b71ec03a" containerName="main"
Oct 31 02:18:37.120716 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.120705    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="d56aacf5-6679-469c-ba41-c68d489ef628" containerName="revoke-keys"
Oct 31 02:18:37.120716 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.120712    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cde8fd2-23bc-46c4-8f56-f6e17339c50d" containerName="main"
Oct 31 02:18:37.125612 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.125588    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359bbaa1-z4nvh"
Oct 31 02:18:37.131041 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.130949    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-359bbaa1-z4nvh"]
Oct 31 02:18:37.132374 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.132348    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:18:37.132546 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.132401    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:18:37.133096 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.132613    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j6ldc\""
Oct 31 02:18:37.134731 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.134709    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:18:37.189535 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.189495    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73bed229-5ad1-415d-99a3-1abba0a327bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-359bbaa1-z4nvh\" (UID: \"73bed229-5ad1-415d-99a3-1abba0a327bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359bbaa1-z4nvh"
Oct 31 02:18:37.189696 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.189556    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4djx4\" (UniqueName: \"kubernetes.io/projected/73bed229-5ad1-415d-99a3-1abba0a327bc-kube-api-access-4djx4\") pod \"maas-api-revoke-sub-359bbaa1-z4nvh\" (UID: \"73bed229-5ad1-415d-99a3-1abba0a327bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359bbaa1-z4nvh"
Oct 31 02:18:37.189696 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.189649    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73bed229-5ad1-415d-99a3-1abba0a327bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-359bbaa1-z4nvh\" (UID: \"73bed229-5ad1-415d-99a3-1abba0a327bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359bbaa1-z4nvh"
Oct 31 02:18:37.290752 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.290711    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73bed229-5ad1-415d-99a3-1abba0a327bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-359bbaa1-z4nvh\" (UID: \"73bed229-5ad1-415d-99a3-1abba0a327bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359bbaa1-z4nvh"
Oct 31 02:18:37.290937 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.290778    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4djx4\" (UniqueName: \"kubernetes.io/projected/73bed229-5ad1-415d-99a3-1abba0a327bc-kube-api-access-4djx4\") pod \"maas-api-revoke-sub-359bbaa1-z4nvh\" (UID: \"73bed229-5ad1-415d-99a3-1abba0a327bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359bbaa1-z4nvh"
Oct 31 02:18:37.290937 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.290799    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73bed229-5ad1-415d-99a3-1abba0a327bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-359bbaa1-z4nvh\" (UID: \"73bed229-5ad1-415d-99a3-1abba0a327bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359bbaa1-z4nvh"
Oct 31 02:18:37.291509 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.291489    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73bed229-5ad1-415d-99a3-1abba0a327bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-359bbaa1-z4nvh\" (UID: \"73bed229-5ad1-415d-99a3-1abba0a327bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359bbaa1-z4nvh"
Oct 31 02:18:37.293357 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.293320    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73bed229-5ad1-415d-99a3-1abba0a327bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-359bbaa1-z4nvh\" (UID: \"73bed229-5ad1-415d-99a3-1abba0a327bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359bbaa1-z4nvh"
Oct 31 02:18:37.306182 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.306147    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4djx4\" (UniqueName: \"kubernetes.io/projected/73bed229-5ad1-415d-99a3-1abba0a327bc-kube-api-access-4djx4\") pod \"maas-api-revoke-sub-359bbaa1-z4nvh\" (UID: \"73bed229-5ad1-415d-99a3-1abba0a327bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359bbaa1-z4nvh"
Oct 31 02:18:37.450043 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.449953    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359bbaa1-z4nvh"
Oct 31 02:18:37.731786 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.731740    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-359bbaa1-z4nvh"]
Oct 31 02:18:37.739098 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:37.739065    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l" podUID="8969a12d-73b4-4d52-a144-918c72ded4bc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:18:38.749963 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:38.749919    5650 generic.go:356] "Generic (PLEG): container finished" podID="73bed229-5ad1-415d-99a3-1abba0a327bc" containerID="63666f71a56771860c7dd78678a79632ef621d8b63303f64c6acc6eadf86159a" exitCode=0
Oct 31 02:18:38.750400 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:38.750009    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359bbaa1-z4nvh" event={"ID":"73bed229-5ad1-415d-99a3-1abba0a327bc","Type":"ContainerDied","Data":"63666f71a56771860c7dd78678a79632ef621d8b63303f64c6acc6eadf86159a"}
Oct 31 02:18:38.750400 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:38.750082    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359bbaa1-z4nvh" event={"ID":"73bed229-5ad1-415d-99a3-1abba0a327bc","Type":"ContainerStarted","Data":"c773e6a2252dfc0636607dc69b53ee628723f51b478505b66a0ffc0599cb2942"}
Oct 31 02:18:39.941757 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:39.941727    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359bbaa1-z4nvh"
Oct 31 02:18:40.048814 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.048787    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73bed229-5ad1-415d-99a3-1abba0a327bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73bed229-5ad1-415d-99a3-1abba0a327bc-maas-api-openshift-ca\") pod \"73bed229-5ad1-415d-99a3-1abba0a327bc\" (UID: \"73bed229-5ad1-415d-99a3-1abba0a327bc\") "
Oct 31 02:18:40.049511 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.048836    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73bed229-5ad1-415d-99a3-1abba0a327bc-maas-api-openshift-ca" pod "73bed229-5ad1-415d-99a3-1abba0a327bc" (UID: "73bed229-5ad1-415d-99a3-1abba0a327bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:40.049761 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.049737    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73bed229-5ad1-415d-99a3-1abba0a327bc-kube-api-access-4djx4\" (UniqueName: \"kubernetes.io/projected/73bed229-5ad1-415d-99a3-1abba0a327bc-kube-api-access-4djx4\") pod \"73bed229-5ad1-415d-99a3-1abba0a327bc\" (UID: \"73bed229-5ad1-415d-99a3-1abba0a327bc\") "
Oct 31 02:18:40.050410 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.050387    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73bed229-5ad1-415d-99a3-1abba0a327bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73bed229-5ad1-415d-99a3-1abba0a327bc-maas-api-xks-ca\") pod \"73bed229-5ad1-415d-99a3-1abba0a327bc\" (UID: \"73bed229-5ad1-415d-99a3-1abba0a327bc\") "
Oct 31 02:18:40.057271 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.057232    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73bed229-5ad1-415d-99a3-1abba0a327bc-maas-api-openshift-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.063111 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.063061    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73bed229-5ad1-415d-99a3-1abba0a327bc-maas-api-xks-ca" pod "73bed229-5ad1-415d-99a3-1abba0a327bc" (UID: "73bed229-5ad1-415d-99a3-1abba0a327bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:18:40.064275 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.064233    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73bed229-5ad1-415d-99a3-1abba0a327bc-kube-api-access-4djx4" pod "73bed229-5ad1-415d-99a3-1abba0a327bc" (UID: "73bed229-5ad1-415d-99a3-1abba0a327bc"). InnerVolumeSpecName "kube-api-access-4djx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:40.159247 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.158700    5650 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73bed229-5ad1-415d-99a3-1abba0a327bc-maas-api-xks-ca\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.159247 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.158738    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4djx4\" (UniqueName: \"kubernetes.io/projected/73bed229-5ad1-415d-99a3-1abba0a327bc-kube-api-access-4djx4\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.401080 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.401046    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 02:18:40.464236 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.464202    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-workload-certs\") pod \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") "
Oct 31 02:18:40.464236 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.464244    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-kube-api-access-5mxjj\" (UniqueName: \"kubernetes.io/projected/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-kube-api-access-5mxjj\") pod \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") "
Oct 31 02:18:40.464483 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.464273    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-envoy\") pod \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") "
Oct 31 02:18:40.464483 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.464324    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-data\") pod \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") "
Oct 31 02:18:40.464483 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.464435    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-token\" (UniqueName: \"kubernetes.io/projected/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-token\") pod \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") "
Oct 31 02:18:40.464581 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.464499    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-credential-socket\") pod \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") "
Oct 31 02:18:40.464581 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.464538    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-workload-socket\") pod \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") "
Oct 31 02:18:40.464581 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.464566    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istiod-ca-cert\") pod \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") "
Oct 31 02:18:40.464731 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.464597    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-podinfo\") pod \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\" (UID: \"bb3603d0-f3c8-4bbc-96c7-366b59bffdce\") "
Oct 31 02:18:40.464731 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.464627    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-workload-certs" pod "bb3603d0-f3c8-4bbc-96c7-366b59bffdce" (UID: "bb3603d0-f3c8-4bbc-96c7-366b59bffdce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:40.465145 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.465091    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-workload-socket" pod "bb3603d0-f3c8-4bbc-96c7-366b59bffdce" (UID: "bb3603d0-f3c8-4bbc-96c7-366b59bffdce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:40.465145 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.465099    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-credential-socket" pod "bb3603d0-f3c8-4bbc-96c7-366b59bffdce" (UID: "bb3603d0-f3c8-4bbc-96c7-366b59bffdce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:40.465384 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.465236    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-data" pod "bb3603d0-f3c8-4bbc-96c7-366b59bffdce" (UID: "bb3603d0-f3c8-4bbc-96c7-366b59bffdce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:40.465384 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.465323    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.465384 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.465348    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.465384 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.465363    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.465931 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.465901    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istiod-ca-cert" pod "bb3603d0-f3c8-4bbc-96c7-366b59bffdce" (UID: "bb3603d0-f3c8-4bbc-96c7-366b59bffdce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:40.466948 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.466924    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-podinfo" pod "bb3603d0-f3c8-4bbc-96c7-366b59bffdce" (UID: "bb3603d0-f3c8-4bbc-96c7-366b59bffdce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:18:40.467192 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.467176    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-envoy" pod "bb3603d0-f3c8-4bbc-96c7-366b59bffdce" (UID: "bb3603d0-f3c8-4bbc-96c7-366b59bffdce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:40.467492 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.467434    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-kube-api-access-5mxjj" pod "bb3603d0-f3c8-4bbc-96c7-366b59bffdce" (UID: "bb3603d0-f3c8-4bbc-96c7-366b59bffdce"). InnerVolumeSpecName "kube-api-access-5mxjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:40.467956 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.467941    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-token" pod "bb3603d0-f3c8-4bbc-96c7-366b59bffdce" (UID: "bb3603d0-f3c8-4bbc-96c7-366b59bffdce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:40.566814 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.566775    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mxjj\" (UniqueName: \"kubernetes.io/projected/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-kube-api-access-5mxjj\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.566814 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.566807    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.566814 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.566817    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.566814 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.566826    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.567062 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.566835    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.567062 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.566843    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb3603d0-f3c8-4bbc-96c7-366b59bffdce-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:40.774077 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.774030    5650 generic.go:356] "Generic (PLEG): container finished" podID="bb3603d0-f3c8-4bbc-96c7-366b59bffdce" containerID="1b460b3b1717dfb6801690ad4515a7f922a63a29cc4c0260e4f3bd715062a724" exitCode=0
Oct 31 02:18:40.774262 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.774100    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"
Oct 31 02:18:40.774262 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.774110    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc" event={"ID":"bb3603d0-f3c8-4bbc-96c7-366b59bffdce","Type":"ContainerDied","Data":"1b460b3b1717dfb6801690ad4515a7f922a63a29cc4c0260e4f3bd715062a724"}
Oct 31 02:18:40.774262 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.774203    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc" event={"ID":"bb3603d0-f3c8-4bbc-96c7-366b59bffdce","Type":"ContainerDied","Data":"1d4953b8738e0ba821cd263ab36e9bf28d9a9ba8a174c4f9fa8107bdb654018f"}
Oct 31 02:18:40.774262 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.774232    5650 scope.go:122] "RemoveContainer" containerID="1b460b3b1717dfb6801690ad4515a7f922a63a29cc4c0260e4f3bd715062a724"
Oct 31 02:18:40.776016 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.775988    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359bbaa1-z4nvh"
Oct 31 02:18:40.776141 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.776029    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359bbaa1-z4nvh" event={"ID":"73bed229-5ad1-415d-99a3-1abba0a327bc","Type":"ContainerDied","Data":"c773e6a2252dfc0636607dc69b53ee628723f51b478505b66a0ffc0599cb2942"}
Oct 31 02:18:40.776141 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.776080    5650 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c773e6a2252dfc0636607dc69b53ee628723f51b478505b66a0ffc0599cb2942"
Oct 31 02:18:40.789035 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.789010    5650 scope.go:122] "RemoveContainer" containerID="1b460b3b1717dfb6801690ad4515a7f922a63a29cc4c0260e4f3bd715062a724"
Oct 31 02:18:40.789696 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:18:40.789665    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b460b3b1717dfb6801690ad4515a7f922a63a29cc4c0260e4f3bd715062a724\": container with ID starting with 1b460b3b1717dfb6801690ad4515a7f922a63a29cc4c0260e4f3bd715062a724 not found: ID does not exist" containerID="1b460b3b1717dfb6801690ad4515a7f922a63a29cc4c0260e4f3bd715062a724"
Oct 31 02:18:40.789794 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.789705    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b460b3b1717dfb6801690ad4515a7f922a63a29cc4c0260e4f3bd715062a724"} err="failed to get container status \"1b460b3b1717dfb6801690ad4515a7f922a63a29cc4c0260e4f3bd715062a724\": rpc error: code = NotFound desc = could not find container \"1b460b3b1717dfb6801690ad4515a7f922a63a29cc4c0260e4f3bd715062a724\": container with ID starting with 1b460b3b1717dfb6801690ad4515a7f922a63a29cc4c0260e4f3bd715062a724 not found: ID does not exist"
Oct 31 02:18:40.803079 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.803048    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"]
Oct 31 02:18:40.806745 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:40.806722    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-28flc"]
Oct 31 02:18:41.206136 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:41.206100    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l"
Oct 31 02:18:41.276606 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:41.276573    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8969a12d-73b4-4d52-a144-918c72ded4bc-kube-api-access-tln6f\" (UniqueName: \"kubernetes.io/projected/8969a12d-73b4-4d52-a144-918c72ded4bc-kube-api-access-tln6f\") pod \"8969a12d-73b4-4d52-a144-918c72ded4bc\" (UID: \"8969a12d-73b4-4d52-a144-918c72ded4bc\") "
Oct 31 02:18:41.276812 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:41.276648    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8969a12d-73b4-4d52-a144-918c72ded4bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8969a12d-73b4-4d52-a144-918c72ded4bc-plugins-config-volume\") pod \"8969a12d-73b4-4d52-a144-918c72ded4bc\" (UID: \"8969a12d-73b4-4d52-a144-918c72ded4bc\") "
Oct 31 02:18:41.277008 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:41.276986    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8969a12d-73b4-4d52-a144-918c72ded4bc-plugins-config-volume" pod "8969a12d-73b4-4d52-a144-918c72ded4bc" (UID: "8969a12d-73b4-4d52-a144-918c72ded4bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:41.278754 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:41.278732    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8969a12d-73b4-4d52-a144-918c72ded4bc-kube-api-access-tln6f" pod "8969a12d-73b4-4d52-a144-918c72ded4bc" (UID: "8969a12d-73b4-4d52-a144-918c72ded4bc"). InnerVolumeSpecName "kube-api-access-tln6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:41.377715 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:41.377629    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tln6f\" (UniqueName: \"kubernetes.io/projected/8969a12d-73b4-4d52-a144-918c72ded4bc-kube-api-access-tln6f\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:41.377715 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:41.377659    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8969a12d-73b4-4d52-a144-918c72ded4bc-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:41.785372 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:41.785334    5650 generic.go:356] "Generic (PLEG): container finished" podID="8969a12d-73b4-4d52-a144-918c72ded4bc" containerID="15ac7aeaaeaadde58c127eb460dccae3f1b72473c0f6086f85769ba1a5e9d298" exitCode=0
Oct 31 02:18:41.785624 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:41.785391    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l"
Oct 31 02:18:41.785624 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:41.785417    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l" event={"ID":"8969a12d-73b4-4d52-a144-918c72ded4bc","Type":"ContainerDied","Data":"15ac7aeaaeaadde58c127eb460dccae3f1b72473c0f6086f85769ba1a5e9d298"}
Oct 31 02:18:41.785624 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:41.785495    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l" event={"ID":"8969a12d-73b4-4d52-a144-918c72ded4bc","Type":"ContainerDied","Data":"8c1b9e02b20a21c9f6561c17bf811260e1501dff1b78a69d635ab7ed9f2041ff"}
Oct 31 02:18:41.785624 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:41.785521    5650 scope.go:122] "RemoveContainer" containerID="15ac7aeaaeaadde58c127eb460dccae3f1b72473c0f6086f85769ba1a5e9d298"
Oct 31 02:18:41.796505 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:41.796481    5650 scope.go:122] "RemoveContainer" containerID="15ac7aeaaeaadde58c127eb460dccae3f1b72473c0f6086f85769ba1a5e9d298"
Oct 31 02:18:41.796845 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:18:41.796821    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15ac7aeaaeaadde58c127eb460dccae3f1b72473c0f6086f85769ba1a5e9d298\": container with ID starting with 15ac7aeaaeaadde58c127eb460dccae3f1b72473c0f6086f85769ba1a5e9d298 not found: ID does not exist" containerID="15ac7aeaaeaadde58c127eb460dccae3f1b72473c0f6086f85769ba1a5e9d298"
Oct 31 02:18:41.796896 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:41.796855    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15ac7aeaaeaadde58c127eb460dccae3f1b72473c0f6086f85769ba1a5e9d298"} err="failed to get container status \"15ac7aeaaeaadde58c127eb460dccae3f1b72473c0f6086f85769ba1a5e9d298\": rpc error: code = NotFound desc = could not find container \"15ac7aeaaeaadde58c127eb460dccae3f1b72473c0f6086f85769ba1a5e9d298\": container with ID starting with 15ac7aeaaeaadde58c127eb460dccae3f1b72473c0f6086f85769ba1a5e9d298 not found: ID does not exist"
Oct 31 02:18:41.806522 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:41.806483    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn" podUID="4640ac90-9987-4201-a94e-56a3102f0f0e" containerName="payload-pre-processing" containerID="cri-o://6ca3c61e7cd270d0a2ca8047384d65b23a7837988c323dcb2a770212b5f71024" gracePeriod=25
Oct 31 02:18:41.816899 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:41.816864    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l"]
Oct 31 02:18:41.821936 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:41.821903    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae5a1c-6d84bc67d7-gj65l"]
Oct 31 02:18:42.020094 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:42.020048    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn" podUID="4640ac90-9987-4201-a94e-56a3102f0f0e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:18:42.026873 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:42.026845    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39c920b7-6kchh"]
Oct 31 02:18:42.030309 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:42.030272    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39c920b7-6kchh"]
Oct 31 02:18:42.117642 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:42.117563    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="014b9592-996c-4034-8453-04e92db14801" path="/var/lib/kubelet/pods/014b9592-996c-4034-8453-04e92db14801/volumes"
Oct 31 02:18:42.117984 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:42.117970    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8969a12d-73b4-4d52-a144-918c72ded4bc" path="/var/lib/kubelet/pods/8969a12d-73b4-4d52-a144-918c72ded4bc/volumes"
Oct 31 02:18:42.118400 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:42.118388    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb3603d0-f3c8-4bbc-96c7-366b59bffdce" path="/var/lib/kubelet/pods/bb3603d0-f3c8-4bbc-96c7-366b59bffdce/volumes"
Oct 31 02:18:42.793675 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:42.793641    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"]
Oct 31 02:18:42.794098 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:42.793853    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7" podUID="dedb244a-f01c-463e-8201-c6d0ad8e3f3f" containerName="istio-proxy" containerID="cri-o://e6a55bdbddfd6bf37a277e04793762730a4c46f074084239cea74340edd8107b" gracePeriod=30
Oct 31 02:18:46.391270 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.391230    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main" containerID="cri-o://884dffa5e0dd7aa2484d580f5630e271197af7425642bbd44cd33097df4b95b6" gracePeriod=45
Oct 31 02:18:46.650897 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.650831    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 02:18:46.771395 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.771363    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a99af64c-978c-4d8c-9d5b-260fc08faf6d-kube-api-access-8whjd\" (UniqueName: \"kubernetes.io/projected/a99af64c-978c-4d8c-9d5b-260fc08faf6d-kube-api-access-8whjd\") pod \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\" (UID: \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\") "
Oct 31 02:18:46.771395 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.771404    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-model-cache\") pod \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\" (UID: \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\") "
Oct 31 02:18:46.771623 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.771430    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-home\" (UniqueName: \"kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-home\") pod \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\" (UID: \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\") "
Oct 31 02:18:46.771623 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.771534    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-tmp-dir\") pod \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\" (UID: \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\") "
Oct 31 02:18:46.771623 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.771612    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a99af64c-978c-4d8c-9d5b-260fc08faf6d-tls-certs\" (UniqueName: \"kubernetes.io/secret/a99af64c-978c-4d8c-9d5b-260fc08faf6d-tls-certs\") pod \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\" (UID: \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\") "
Oct 31 02:18:46.771774 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.771659    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-dshm\") pod \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\" (UID: \"a99af64c-978c-4d8c-9d5b-260fc08faf6d\") "
Oct 31 02:18:46.772175 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.771808    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-model-cache" pod "a99af64c-978c-4d8c-9d5b-260fc08faf6d" (UID: "a99af64c-978c-4d8c-9d5b-260fc08faf6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:46.772175 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.771821    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-home" pod "a99af64c-978c-4d8c-9d5b-260fc08faf6d" (UID: "a99af64c-978c-4d8c-9d5b-260fc08faf6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:46.772175 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.771889    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-tmp-dir" pod "a99af64c-978c-4d8c-9d5b-260fc08faf6d" (UID: "a99af64c-978c-4d8c-9d5b-260fc08faf6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:46.772175 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.772127    5650 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-tmp-dir\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:46.772175 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.772148    5650 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-model-cache\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:46.772175 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.772158    5650 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-home\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:46.773967 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.773942    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-dshm" pod "a99af64c-978c-4d8c-9d5b-260fc08faf6d" (UID: "a99af64c-978c-4d8c-9d5b-260fc08faf6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:46.773967 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.773957    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a99af64c-978c-4d8c-9d5b-260fc08faf6d-tls-certs" pod "a99af64c-978c-4d8c-9d5b-260fc08faf6d" (UID: "a99af64c-978c-4d8c-9d5b-260fc08faf6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:18:46.774163 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.774000    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99af64c-978c-4d8c-9d5b-260fc08faf6d-kube-api-access-8whjd" pod "a99af64c-978c-4d8c-9d5b-260fc08faf6d" (UID: "a99af64c-978c-4d8c-9d5b-260fc08faf6d"). InnerVolumeSpecName "kube-api-access-8whjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:46.824823 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.824784    5650 generic.go:356] "Generic (PLEG): container finished" podID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerID="884dffa5e0dd7aa2484d580f5630e271197af7425642bbd44cd33097df4b95b6" exitCode=0
Oct 31 02:18:46.825006 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.824830    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"
Oct 31 02:18:46.825006 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.824836    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" event={"ID":"a99af64c-978c-4d8c-9d5b-260fc08faf6d","Type":"ContainerDied","Data":"884dffa5e0dd7aa2484d580f5630e271197af7425642bbd44cd33097df4b95b6"}
Oct 31 02:18:46.825006 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.824875    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth" event={"ID":"a99af64c-978c-4d8c-9d5b-260fc08faf6d","Type":"ContainerDied","Data":"cca8747a34fa20f08cafc5b7f873ef7befff4b5caa7608a7df6273a8ad0bff5f"}
Oct 31 02:18:46.825006 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.824897    5650 scope.go:122] "RemoveContainer" containerID="884dffa5e0dd7aa2484d580f5630e271197af7425642bbd44cd33097df4b95b6"
Oct 31 02:18:46.837216 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.837189    5650 scope.go:122] "RemoveContainer" containerID="884dffa5e0dd7aa2484d580f5630e271197af7425642bbd44cd33097df4b95b6"
Oct 31 02:18:46.837575 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:18:46.837554    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"884dffa5e0dd7aa2484d580f5630e271197af7425642bbd44cd33097df4b95b6\": container with ID starting with 884dffa5e0dd7aa2484d580f5630e271197af7425642bbd44cd33097df4b95b6 not found: ID does not exist" containerID="884dffa5e0dd7aa2484d580f5630e271197af7425642bbd44cd33097df4b95b6"
Oct 31 02:18:46.837667 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.837585    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"884dffa5e0dd7aa2484d580f5630e271197af7425642bbd44cd33097df4b95b6"} err="failed to get container status \"884dffa5e0dd7aa2484d580f5630e271197af7425642bbd44cd33097df4b95b6\": rpc error: code = NotFound desc = could not find container \"884dffa5e0dd7aa2484d580f5630e271197af7425642bbd44cd33097df4b95b6\": container with ID starting with 884dffa5e0dd7aa2484d580f5630e271197af7425642bbd44cd33097df4b95b6 not found: ID does not exist"
Oct 31 02:18:46.850941 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.850914    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"]
Oct 31 02:18:46.857646 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.857620    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-7e5955/premium-w3-7e5955-kserve-55f747475-rzgth"]
Oct 31 02:18:46.872761 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.872731    5650 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a99af64c-978c-4d8c-9d5b-260fc08faf6d-tls-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:46.872761 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.872760    5650 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a99af64c-978c-4d8c-9d5b-260fc08faf6d-dshm\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:46.872946 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:46.872774    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8whjd\" (UniqueName: \"kubernetes.io/projected/a99af64c-978c-4d8c-9d5b-260fc08faf6d-kube-api-access-8whjd\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:47.192322 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:47.192286    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"]
Oct 31 02:18:47.192578 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:47.192534    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h" podUID="f14608d5-81ae-4ba3-8986-a8e602befe12" containerName="istio-proxy" containerID="cri-o://5f2902debda7847f750050fc5485f68e6a58eeaecc2a246cddf916fd5923ad9f" gracePeriod=30
Oct 31 02:18:48.060441 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.060376    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 02:18:48.120071 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.120035    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" path="/var/lib/kubelet/pods/a99af64c-978c-4d8c-9d5b-260fc08faf6d/volumes"
Oct 31 02:18:48.190349 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.190310    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-envoy\") pod \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") "
Oct 31 02:18:48.190594 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.190408    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-workload-socket\") pod \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") "
Oct 31 02:18:48.190594 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.190471    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-token\" (UniqueName: \"kubernetes.io/projected/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-token\") pod \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") "
Oct 31 02:18:48.190594 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.190511    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-data\") pod \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") "
Oct 31 02:18:48.190594 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.190588    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-workload-certs\") pod \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") "
Oct 31 02:18:48.190839 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.190623    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istiod-ca-cert\") pod \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") "
Oct 31 02:18:48.190839 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.190663    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-credential-socket\") pod \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") "
Oct 31 02:18:48.190839 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.190696    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-workload-socket" pod "dedb244a-f01c-463e-8201-c6d0ad8e3f3f" (UID: "dedb244a-f01c-463e-8201-c6d0ad8e3f3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:48.190839 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.190704    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-kube-api-access-cmg97\" (UniqueName: \"kubernetes.io/projected/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-kube-api-access-cmg97\") pod \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") "
Oct 31 02:18:48.190839 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.190825    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-podinfo\") pod \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\" (UID: \"dedb244a-f01c-463e-8201-c6d0ad8e3f3f\") "
Oct 31 02:18:48.191110 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.191058    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-workload-certs" pod "dedb244a-f01c-463e-8201-c6d0ad8e3f3f" (UID: "dedb244a-f01c-463e-8201-c6d0ad8e3f3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:48.191465 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.191420    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-data" pod "dedb244a-f01c-463e-8201-c6d0ad8e3f3f" (UID: "dedb244a-f01c-463e-8201-c6d0ad8e3f3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:48.191650 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.191467    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:48.191650 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.191486    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:48.191903 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.191723    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istiod-ca-cert" pod "dedb244a-f01c-463e-8201-c6d0ad8e3f3f" (UID: "dedb244a-f01c-463e-8201-c6d0ad8e3f3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:48.192036 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.192020    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-credential-socket" pod "dedb244a-f01c-463e-8201-c6d0ad8e3f3f" (UID: "dedb244a-f01c-463e-8201-c6d0ad8e3f3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:48.193137 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.193111    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-envoy" pod "dedb244a-f01c-463e-8201-c6d0ad8e3f3f" (UID: "dedb244a-f01c-463e-8201-c6d0ad8e3f3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:48.193422 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.193395    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-kube-api-access-cmg97" pod "dedb244a-f01c-463e-8201-c6d0ad8e3f3f" (UID: "dedb244a-f01c-463e-8201-c6d0ad8e3f3f"). InnerVolumeSpecName "kube-api-access-cmg97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:48.193753 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.193730    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-token" pod "dedb244a-f01c-463e-8201-c6d0ad8e3f3f" (UID: "dedb244a-f01c-463e-8201-c6d0ad8e3f3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:48.194353 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.194332    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-podinfo" pod "dedb244a-f01c-463e-8201-c6d0ad8e3f3f" (UID: "dedb244a-f01c-463e-8201-c6d0ad8e3f3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:18:48.292342 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.292294    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:48.292342 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.292333    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:48.292342 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.292344    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmg97\" (UniqueName: \"kubernetes.io/projected/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-kube-api-access-cmg97\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:48.292342 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.292353    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:48.292673 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.292362    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:48.292673 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.292371    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:48.292673 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.292379    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dedb244a-f01c-463e-8201-c6d0ad8e3f3f-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:48.667992 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.667960    5650 scope.go:122] "RemoveContainer" containerID="e6a55bdbddfd6bf37a277e04793762730a4c46f074084239cea74340edd8107b"
Oct 31 02:18:48.838852 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.838814    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"
Oct 31 02:18:48.839023 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.838814    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7" event={"ID":"dedb244a-f01c-463e-8201-c6d0ad8e3f3f","Type":"ContainerDied","Data":"e6a55bdbddfd6bf37a277e04793762730a4c46f074084239cea74340edd8107b"}
Oct 31 02:18:48.839023 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.838983    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7" event={"ID":"dedb244a-f01c-463e-8201-c6d0ad8e3f3f","Type":"ContainerDied","Data":"aba6fbb2f54c7f5414a9318003a49e72d1d845deba71a8a330953315b765254a"}
Oct 31 02:18:48.891713 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.891677    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"]
Oct 31 02:18:48.896036 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:48.895997    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kf8f7"]
Oct 31 02:18:49.891530 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:49.891497    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf"]
Oct 31 02:18:49.897180 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:49.897141    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-7e5955-wrpbf"]
Oct 31 02:18:50.117227 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:50.117194    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d56aacf5-6679-469c-ba41-c68d489ef628" path="/var/lib/kubelet/pods/d56aacf5-6679-469c-ba41-c68d489ef628/volumes"
Oct 31 02:18:50.117672 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:50.117656    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dedb244a-f01c-463e-8201-c6d0ad8e3f3f" path="/var/lib/kubelet/pods/dedb244a-f01c-463e-8201-c6d0ad8e3f3f/volumes"
Oct 31 02:18:52.020116 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.020071    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn" podUID="4640ac90-9987-4201-a94e-56a3102f0f0e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:18:52.454105 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.454080    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 02:18:52.643768 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.643675    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-credential-socket\") pod \"f14608d5-81ae-4ba3-8986-a8e602befe12\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") "
Oct 31 02:18:52.643768 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.643766    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-podinfo\") pod \"f14608d5-81ae-4ba3-8986-a8e602befe12\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") "
Oct 31 02:18:52.643995 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.643800    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-workload-socket\") pod \"f14608d5-81ae-4ba3-8986-a8e602befe12\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") "
Oct 31 02:18:52.643995 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.643824    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-token\" (UniqueName: \"kubernetes.io/projected/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-token\") pod \"f14608d5-81ae-4ba3-8986-a8e602befe12\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") "
Oct 31 02:18:52.643995 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.643862    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-workload-certs\") pod \"f14608d5-81ae-4ba3-8986-a8e602befe12\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") "
Oct 31 02:18:52.643995 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.643903    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f14608d5-81ae-4ba3-8986-a8e602befe12-kube-api-access-rcghd\" (UniqueName: \"kubernetes.io/projected/f14608d5-81ae-4ba3-8986-a8e602befe12-kube-api-access-rcghd\") pod \"f14608d5-81ae-4ba3-8986-a8e602befe12\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") "
Oct 31 02:18:52.643995 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.643956    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-envoy\") pod \"f14608d5-81ae-4ba3-8986-a8e602befe12\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") "
Oct 31 02:18:52.644267 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.644000    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f14608d5-81ae-4ba3-8986-a8e602befe12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f14608d5-81ae-4ba3-8986-a8e602befe12-istiod-ca-cert\") pod \"f14608d5-81ae-4ba3-8986-a8e602befe12\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") "
Oct 31 02:18:52.644267 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.644030    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-credential-socket" pod "f14608d5-81ae-4ba3-8986-a8e602befe12" (UID: "f14608d5-81ae-4ba3-8986-a8e602befe12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:52.644267 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.644063    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-data\") pod \"f14608d5-81ae-4ba3-8986-a8e602befe12\" (UID: \"f14608d5-81ae-4ba3-8986-a8e602befe12\") "
Oct 31 02:18:52.644267 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.644150    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-workload-socket" pod "f14608d5-81ae-4ba3-8986-a8e602befe12" (UID: "f14608d5-81ae-4ba3-8986-a8e602befe12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:52.644548 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.644318    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-workload-certs" pod "f14608d5-81ae-4ba3-8986-a8e602befe12" (UID: "f14608d5-81ae-4ba3-8986-a8e602befe12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:52.644548 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.644382    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-data" pod "f14608d5-81ae-4ba3-8986-a8e602befe12" (UID: "f14608d5-81ae-4ba3-8986-a8e602befe12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:52.644548 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.644474    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f14608d5-81ae-4ba3-8986-a8e602befe12-istiod-ca-cert" pod "f14608d5-81ae-4ba3-8986-a8e602befe12" (UID: "f14608d5-81ae-4ba3-8986-a8e602befe12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:18:52.644726 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.644580    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:52.644726 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.644600    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:52.644726 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.644616    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f14608d5-81ae-4ba3-8986-a8e602befe12-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:52.644726 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.644630    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:52.644726 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.644645    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:52.646190 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.646162    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-podinfo" pod "f14608d5-81ae-4ba3-8986-a8e602befe12" (UID: "f14608d5-81ae-4ba3-8986-a8e602befe12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:18:52.646299 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.646189    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f14608d5-81ae-4ba3-8986-a8e602befe12-kube-api-access-rcghd" pod "f14608d5-81ae-4ba3-8986-a8e602befe12" (UID: "f14608d5-81ae-4ba3-8986-a8e602befe12"). InnerVolumeSpecName "kube-api-access-rcghd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:52.646484 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.646458    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-token" pod "f14608d5-81ae-4ba3-8986-a8e602befe12" (UID: "f14608d5-81ae-4ba3-8986-a8e602befe12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:18:52.646607 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.646593    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-envoy" pod "f14608d5-81ae-4ba3-8986-a8e602befe12" (UID: "f14608d5-81ae-4ba3-8986-a8e602befe12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:18:52.746050 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.746011    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:52.746050 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.746044    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:52.746050 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.746054    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcghd\" (UniqueName: \"kubernetes.io/projected/f14608d5-81ae-4ba3-8986-a8e602befe12-kube-api-access-rcghd\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:52.746313 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.746064    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f14608d5-81ae-4ba3-8986-a8e602befe12-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:18:52.862404 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.862373    5650 generic.go:356] "Generic (PLEG): container finished" podID="f14608d5-81ae-4ba3-8986-a8e602befe12" containerID="5f2902debda7847f750050fc5485f68e6a58eeaecc2a246cddf916fd5923ad9f" exitCode=0
Oct 31 02:18:52.862647 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.862420    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"
Oct 31 02:18:52.862647 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.862488    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h" event={"ID":"f14608d5-81ae-4ba3-8986-a8e602befe12","Type":"ContainerDied","Data":"5f2902debda7847f750050fc5485f68e6a58eeaecc2a246cddf916fd5923ad9f"}
Oct 31 02:18:52.862647 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.862550    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h" event={"ID":"f14608d5-81ae-4ba3-8986-a8e602befe12","Type":"ContainerDied","Data":"8bdc075509e508c2d8809b35bc2cae54d87e3c83a6dfe44e124ff921444bd5b0"}
Oct 31 02:18:52.862647 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.862583    5650 scope.go:122] "RemoveContainer" containerID="5f2902debda7847f750050fc5485f68e6a58eeaecc2a246cddf916fd5923ad9f"
Oct 31 02:18:52.873257 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.873236    5650 scope.go:122] "RemoveContainer" containerID="5f2902debda7847f750050fc5485f68e6a58eeaecc2a246cddf916fd5923ad9f"
Oct 31 02:18:52.873567 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:18:52.873548    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f2902debda7847f750050fc5485f68e6a58eeaecc2a246cddf916fd5923ad9f\": container with ID starting with 5f2902debda7847f750050fc5485f68e6a58eeaecc2a246cddf916fd5923ad9f not found: ID does not exist" containerID="5f2902debda7847f750050fc5485f68e6a58eeaecc2a246cddf916fd5923ad9f"
Oct 31 02:18:52.873626 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.873577    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f2902debda7847f750050fc5485f68e6a58eeaecc2a246cddf916fd5923ad9f"} err="failed to get container status \"5f2902debda7847f750050fc5485f68e6a58eeaecc2a246cddf916fd5923ad9f\": rpc error: code = NotFound desc = could not find container \"5f2902debda7847f750050fc5485f68e6a58eeaecc2a246cddf916fd5923ad9f\": container with ID starting with 5f2902debda7847f750050fc5485f68e6a58eeaecc2a246cddf916fd5923ad9f not found: ID does not exist"
Oct 31 02:18:52.892775 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.892745    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"]
Oct 31 02:18:52.897549 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:52.897469    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-7rq2h"]
Oct 31 02:18:54.117083 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:54.117048    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f14608d5-81ae-4ba3-8986-a8e602befe12" path="/var/lib/kubelet/pods/f14608d5-81ae-4ba3-8986-a8e602befe12/volumes"
Oct 31 02:18:58.388983 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:58.388933    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"]
Oct 31 02:18:58.390161 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:58.389187    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh" podUID="da800208-fbd5-460f-b1cc-a212a4445291" containerName="istio-proxy" containerID="cri-o://43b83ebd4624f35d9ba435a41f1d41d2f23b44cd8c5dce683dc9aeca5732f362" gracePeriod=30
Oct 31 02:18:58.393213 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:58.393185    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"]
Oct 31 02:18:58.393436 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:58.393412    5650 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8" podUID="af0ee471-3ebf-426f-88e8-689ddc123d8e" containerName="istio-proxy" containerID="cri-o://e922da61c00fc4f4fc74f93416db756b022d7165e2f7329716954140e3d6d8a2" gracePeriod=30
Oct 31 02:18:58.553580 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:18:58.553540    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:18:59.625979 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:18:59.625941    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-544f7c9987-fwhzc_f0f46d02-6b50-40ec-b940-f767a9ce2b72/authorino/0.log"
Oct 31 02:19:02.089644 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:02.089612    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn"
Oct 31 02:19:02.149876 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:02.149837    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4640ac90-9987-4201-a94e-56a3102f0f0e-kube-api-access-9rttc\" (UniqueName: \"kubernetes.io/projected/4640ac90-9987-4201-a94e-56a3102f0f0e-kube-api-access-9rttc\") pod \"4640ac90-9987-4201-a94e-56a3102f0f0e\" (UID: \"4640ac90-9987-4201-a94e-56a3102f0f0e\") "
Oct 31 02:19:02.150077 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:02.149936    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4640ac90-9987-4201-a94e-56a3102f0f0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4640ac90-9987-4201-a94e-56a3102f0f0e-plugins-config-volume\") pod \"4640ac90-9987-4201-a94e-56a3102f0f0e\" (UID: \"4640ac90-9987-4201-a94e-56a3102f0f0e\") "
Oct 31 02:19:02.150361 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:02.150338    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4640ac90-9987-4201-a94e-56a3102f0f0e-plugins-config-volume" pod "4640ac90-9987-4201-a94e-56a3102f0f0e" (UID: "4640ac90-9987-4201-a94e-56a3102f0f0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:19:02.150439 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:02.150423    5650 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4640ac90-9987-4201-a94e-56a3102f0f0e-plugins-config-volume\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:02.152230 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:02.152198    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4640ac90-9987-4201-a94e-56a3102f0f0e-kube-api-access-9rttc" pod "4640ac90-9987-4201-a94e-56a3102f0f0e" (UID: "4640ac90-9987-4201-a94e-56a3102f0f0e"). InnerVolumeSpecName "kube-api-access-9rttc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:19:02.207335 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:02.207303    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-75686f85d5-6f7td_4e714053-4cce-4192-a7a3-cc1ddb6c25fc/manager/0.log"
Oct 31 02:19:02.251308 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:02.251269    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rttc\" (UniqueName: \"kubernetes.io/projected/4640ac90-9987-4201-a94e-56a3102f0f0e-kube-api-access-9rttc\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:02.923582 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:02.923542    5650 generic.go:356] "Generic (PLEG): container finished" podID="4640ac90-9987-4201-a94e-56a3102f0f0e" containerID="6ca3c61e7cd270d0a2ca8047384d65b23a7837988c323dcb2a770212b5f71024" exitCode=0
Oct 31 02:19:02.923774 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:02.923594    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn"
Oct 31 02:19:02.923774 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:02.923628    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn" event={"ID":"4640ac90-9987-4201-a94e-56a3102f0f0e","Type":"ContainerDied","Data":"6ca3c61e7cd270d0a2ca8047384d65b23a7837988c323dcb2a770212b5f71024"}
Oct 31 02:19:02.923774 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:02.923685    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn" event={"ID":"4640ac90-9987-4201-a94e-56a3102f0f0e","Type":"ContainerDied","Data":"90d480a80cd7e0ba7a121c0083487fd91d01fe4bd83f0a79a4ceacfbff37dc07"}
Oct 31 02:19:02.923774 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:02.923707    5650 scope.go:122] "RemoveContainer" containerID="6ca3c61e7cd270d0a2ca8047384d65b23a7837988c323dcb2a770212b5f71024"
Oct 31 02:19:02.934673 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:02.934650    5650 scope.go:122] "RemoveContainer" containerID="6ca3c61e7cd270d0a2ca8047384d65b23a7837988c323dcb2a770212b5f71024"
Oct 31 02:19:02.934974 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:19:02.934955    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ca3c61e7cd270d0a2ca8047384d65b23a7837988c323dcb2a770212b5f71024\": container with ID starting with 6ca3c61e7cd270d0a2ca8047384d65b23a7837988c323dcb2a770212b5f71024 not found: ID does not exist" containerID="6ca3c61e7cd270d0a2ca8047384d65b23a7837988c323dcb2a770212b5f71024"
Oct 31 02:19:02.935020 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:02.934985    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ca3c61e7cd270d0a2ca8047384d65b23a7837988c323dcb2a770212b5f71024"} err="failed to get container status \"6ca3c61e7cd270d0a2ca8047384d65b23a7837988c323dcb2a770212b5f71024\": rpc error: code = NotFound desc = could not find container \"6ca3c61e7cd270d0a2ca8047384d65b23a7837988c323dcb2a770212b5f71024\": container with ID starting with 6ca3c61e7cd270d0a2ca8047384d65b23a7837988c323dcb2a770212b5f71024 not found: ID does not exist"
Oct 31 02:19:02.948538 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:02.948515    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn"]
Oct 31 02:19:02.952766 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:02.952742    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn"]
Oct 31 02:19:03.019686 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.019498    5650 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-7e5955-7c6f68ff47-p5cwn" podUID="4640ac90-9987-4201-a94e-56a3102f0f0e" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.59:50052\" within 1s: context deadline exceeded"
Oct 31 02:19:03.019686 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:19:03.019641    5650 logging.go:55] [core] [Channel #10610 SubChannel #10611]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.59:50052", ServerName: "10.134.0.59:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.59:50052: operation was canceled"
Oct 31 02:19:03.654872 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.654843    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 02:19:03.690867 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.690779    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 02:19:03.767039 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.767003    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-istio-data\") pod \"da800208-fbd5-460f-b1cc-a212a4445291\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") "
Oct 31 02:19:03.767250 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.767050    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-envoy\") pod \"af0ee471-3ebf-426f-88e8-689ddc123d8e\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") "
Oct 31 02:19:03.767250 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.767091    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da800208-fbd5-460f-b1cc-a212a4445291-kube-api-access-rnpvm\" (UniqueName: \"kubernetes.io/projected/da800208-fbd5-460f-b1cc-a212a4445291-kube-api-access-rnpvm\") pod \"da800208-fbd5-460f-b1cc-a212a4445291\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") "
Oct 31 02:19:03.767250 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.767120    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da800208-fbd5-460f-b1cc-a212a4445291-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da800208-fbd5-460f-b1cc-a212a4445291-istiod-ca-cert\") pod \"da800208-fbd5-460f-b1cc-a212a4445291\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") "
Oct 31 02:19:03.767250 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.767188    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-credential-socket\") pod \"da800208-fbd5-460f-b1cc-a212a4445291\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") "
Oct 31 02:19:03.767250 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.767219    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-credential-socket\") pod \"af0ee471-3ebf-426f-88e8-689ddc123d8e\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") "
Oct 31 02:19:03.767523 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.767256    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-workload-certs\") pod \"af0ee471-3ebf-426f-88e8-689ddc123d8e\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") "
Oct 31 02:19:03.767523 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.767310    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-token\" (UniqueName: \"kubernetes.io/projected/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-token\") pod \"af0ee471-3ebf-426f-88e8-689ddc123d8e\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") "
Oct 31 02:19:03.767523 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.767338    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da800208-fbd5-460f-b1cc-a212a4445291-istio-token\" (UniqueName: \"kubernetes.io/projected/da800208-fbd5-460f-b1cc-a212a4445291-istio-token\") pod \"da800208-fbd5-460f-b1cc-a212a4445291\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") "
Oct 31 02:19:03.767523 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.767367    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af0ee471-3ebf-426f-88e8-689ddc123d8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af0ee471-3ebf-426f-88e8-689ddc123d8e-istiod-ca-cert\") pod \"af0ee471-3ebf-426f-88e8-689ddc123d8e\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") "
Oct 31 02:19:03.767523 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.767396    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-workload-certs\") pod \"da800208-fbd5-460f-b1cc-a212a4445291\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") "
Oct 31 02:19:03.767523 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.767478    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da800208-fbd5-460f-b1cc-a212a4445291-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da800208-fbd5-460f-b1cc-a212a4445291-istio-podinfo\") pod \"da800208-fbd5-460f-b1cc-a212a4445291\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") "
Oct 31 02:19:03.767523 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.767515    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-workload-socket\") pod \"da800208-fbd5-460f-b1cc-a212a4445291\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") "
Oct 31 02:19:03.767774 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.767550    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-workload-socket\") pod \"af0ee471-3ebf-426f-88e8-689ddc123d8e\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") "
Oct 31 02:19:03.767774 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.767573    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-data\") pod \"af0ee471-3ebf-426f-88e8-689ddc123d8e\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") "
Oct 31 02:19:03.767774 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.767595    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-podinfo\") pod \"af0ee471-3ebf-426f-88e8-689ddc123d8e\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") "
Oct 31 02:19:03.767774 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.767622    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-istio-envoy\") pod \"da800208-fbd5-460f-b1cc-a212a4445291\" (UID: \"da800208-fbd5-460f-b1cc-a212a4445291\") "
Oct 31 02:19:03.767774 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.767700    5650 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af0ee471-3ebf-426f-88e8-689ddc123d8e-kube-api-access-w979g\" (UniqueName: \"kubernetes.io/projected/af0ee471-3ebf-426f-88e8-689ddc123d8e-kube-api-access-w979g\") pod \"af0ee471-3ebf-426f-88e8-689ddc123d8e\" (UID: \"af0ee471-3ebf-426f-88e8-689ddc123d8e\") "
Oct 31 02:19:03.791796 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.791682    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-istio-data" pod "da800208-fbd5-460f-b1cc-a212a4445291" (UID: "da800208-fbd5-460f-b1cc-a212a4445291"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:19:03.792023 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.791902    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-workload-certs" pod "da800208-fbd5-460f-b1cc-a212a4445291" (UID: "da800208-fbd5-460f-b1cc-a212a4445291"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:19:03.792023 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.791930    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-envoy" pod "af0ee471-3ebf-426f-88e8-689ddc123d8e" (UID: "af0ee471-3ebf-426f-88e8-689ddc123d8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:19:03.792165 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.792033    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-data" pod "af0ee471-3ebf-426f-88e8-689ddc123d8e" (UID: "af0ee471-3ebf-426f-88e8-689ddc123d8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:19:03.792165 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.792114    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-workload-socket" pod "af0ee471-3ebf-426f-88e8-689ddc123d8e" (UID: "af0ee471-3ebf-426f-88e8-689ddc123d8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:19:03.792165 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.792119    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-workload-socket" pod "da800208-fbd5-460f-b1cc-a212a4445291" (UID: "da800208-fbd5-460f-b1cc-a212a4445291"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:19:03.792888 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.792238    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-credential-socket" pod "da800208-fbd5-460f-b1cc-a212a4445291" (UID: "da800208-fbd5-460f-b1cc-a212a4445291"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:19:03.792888 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.792250    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da800208-fbd5-460f-b1cc-a212a4445291-kube-api-access-rnpvm" pod "da800208-fbd5-460f-b1cc-a212a4445291" (UID: "da800208-fbd5-460f-b1cc-a212a4445291"). InnerVolumeSpecName "kube-api-access-rnpvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:19:03.792888 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.792291    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-credential-socket" pod "af0ee471-3ebf-426f-88e8-689ddc123d8e" (UID: "af0ee471-3ebf-426f-88e8-689ddc123d8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:19:03.792888 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.792421    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af0ee471-3ebf-426f-88e8-689ddc123d8e-istiod-ca-cert" pod "af0ee471-3ebf-426f-88e8-689ddc123d8e" (UID: "af0ee471-3ebf-426f-88e8-689ddc123d8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:19:03.792888 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.792530    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da800208-fbd5-460f-b1cc-a212a4445291-istiod-ca-cert" pod "da800208-fbd5-460f-b1cc-a212a4445291" (UID: "da800208-fbd5-460f-b1cc-a212a4445291"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:19:03.792888 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.792582    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-workload-certs" pod "af0ee471-3ebf-426f-88e8-689ddc123d8e" (UID: "af0ee471-3ebf-426f-88e8-689ddc123d8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:19:03.793985 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.793958    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-istio-envoy" pod "da800208-fbd5-460f-b1cc-a212a4445291" (UID: "da800208-fbd5-460f-b1cc-a212a4445291"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:19:03.794092 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.794035    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da800208-fbd5-460f-b1cc-a212a4445291-istio-token" pod "da800208-fbd5-460f-b1cc-a212a4445291" (UID: "da800208-fbd5-460f-b1cc-a212a4445291"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:19:03.794173 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.794154    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da800208-fbd5-460f-b1cc-a212a4445291-istio-podinfo" pod "da800208-fbd5-460f-b1cc-a212a4445291" (UID: "da800208-fbd5-460f-b1cc-a212a4445291"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:19:03.794370 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.794356    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-token" pod "af0ee471-3ebf-426f-88e8-689ddc123d8e" (UID: "af0ee471-3ebf-426f-88e8-689ddc123d8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:19:03.794837 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.794806    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af0ee471-3ebf-426f-88e8-689ddc123d8e-kube-api-access-w979g" pod "af0ee471-3ebf-426f-88e8-689ddc123d8e" (UID: "af0ee471-3ebf-426f-88e8-689ddc123d8e"). InnerVolumeSpecName "kube-api-access-w979g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:19:03.794837 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.794809    5650 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-podinfo" pod "af0ee471-3ebf-426f-88e8-689ddc123d8e" (UID: "af0ee471-3ebf-426f-88e8-689ddc123d8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:19:03.868808 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.868777    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w979g\" (UniqueName: \"kubernetes.io/projected/af0ee471-3ebf-426f-88e8-689ddc123d8e-kube-api-access-w979g\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.868808 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.868805    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.868997 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.868816    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.868997 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.868830    5650 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnpvm\" (UniqueName: \"kubernetes.io/projected/da800208-fbd5-460f-b1cc-a212a4445291-kube-api-access-rnpvm\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.868997 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.868843    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da800208-fbd5-460f-b1cc-a212a4445291-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.868997 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.868852    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.868997 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.868861    5650 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-credential-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.868997 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.868869    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.868997 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.868877    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.868997 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.868886    5650 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da800208-fbd5-460f-b1cc-a212a4445291-istio-token\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.868997 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.868897    5650 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af0ee471-3ebf-426f-88e8-689ddc123d8e-istiod-ca-cert\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.868997 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.868912    5650 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-workload-certs\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.868997 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.868923    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da800208-fbd5-460f-b1cc-a212a4445291-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.868997 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.868932    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.868997 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.868941    5650 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-workload-socket\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.868997 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.868949    5650 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-data\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.868997 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.868958    5650 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af0ee471-3ebf-426f-88e8-689ddc123d8e-istio-podinfo\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.868997 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.868966    5650 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da800208-fbd5-460f-b1cc-a212a4445291-istio-envoy\") on node \"ip-10-0-133-98.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 02:19:03.931355 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.931266    5650 generic.go:356] "Generic (PLEG): container finished" podID="af0ee471-3ebf-426f-88e8-689ddc123d8e" containerID="e922da61c00fc4f4fc74f93416db756b022d7165e2f7329716954140e3d6d8a2" exitCode=0
Oct 31 02:19:03.931355 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.931314    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"
Oct 31 02:19:03.931623 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.931353    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8" event={"ID":"af0ee471-3ebf-426f-88e8-689ddc123d8e","Type":"ContainerDied","Data":"e922da61c00fc4f4fc74f93416db756b022d7165e2f7329716954140e3d6d8a2"}
Oct 31 02:19:03.931623 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.931415    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8" event={"ID":"af0ee471-3ebf-426f-88e8-689ddc123d8e","Type":"ContainerDied","Data":"4ed865c42a4f2fca8fed7bda29887bd30f117133ae02b03c07b88aa3aebd411a"}
Oct 31 02:19:03.931623 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.931436    5650 scope.go:122] "RemoveContainer" containerID="e922da61c00fc4f4fc74f93416db756b022d7165e2f7329716954140e3d6d8a2"
Oct 31 02:19:03.933815 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.933788    5650 generic.go:356] "Generic (PLEG): container finished" podID="da800208-fbd5-460f-b1cc-a212a4445291" containerID="43b83ebd4624f35d9ba435a41f1d41d2f23b44cd8c5dce683dc9aeca5732f362" exitCode=0
Oct 31 02:19:03.933954 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.933842    5650 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"
Oct 31 02:19:03.933954 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.933869    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh" event={"ID":"da800208-fbd5-460f-b1cc-a212a4445291","Type":"ContainerDied","Data":"43b83ebd4624f35d9ba435a41f1d41d2f23b44cd8c5dce683dc9aeca5732f362"}
Oct 31 02:19:03.933954 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.933934    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh" event={"ID":"da800208-fbd5-460f-b1cc-a212a4445291","Type":"ContainerDied","Data":"bc929d49a9b4a1f807e39953fb2d7ed2e8d7e63a2e750f77fb486fa1b02830d2"}
Oct 31 02:19:03.942429 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.942409    5650 scope.go:122] "RemoveContainer" containerID="fe2949ea3820d9e9e31cc129190a905dc7b124e5b6c3d8528eac8f17423881f1"
Oct 31 02:19:03.951469 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.951431    5650 scope.go:122] "RemoveContainer" containerID="e922da61c00fc4f4fc74f93416db756b022d7165e2f7329716954140e3d6d8a2"
Oct 31 02:19:03.951761 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:19:03.951742    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e922da61c00fc4f4fc74f93416db756b022d7165e2f7329716954140e3d6d8a2\": container with ID starting with e922da61c00fc4f4fc74f93416db756b022d7165e2f7329716954140e3d6d8a2 not found: ID does not exist" containerID="e922da61c00fc4f4fc74f93416db756b022d7165e2f7329716954140e3d6d8a2"
Oct 31 02:19:03.951842 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.951774    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e922da61c00fc4f4fc74f93416db756b022d7165e2f7329716954140e3d6d8a2"} err="failed to get container status \"e922da61c00fc4f4fc74f93416db756b022d7165e2f7329716954140e3d6d8a2\": rpc error: code = NotFound desc = could not find container \"e922da61c00fc4f4fc74f93416db756b022d7165e2f7329716954140e3d6d8a2\": container with ID starting with e922da61c00fc4f4fc74f93416db756b022d7165e2f7329716954140e3d6d8a2 not found: ID does not exist"
Oct 31 02:19:03.951842 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.951803    5650 scope.go:122] "RemoveContainer" containerID="fe2949ea3820d9e9e31cc129190a905dc7b124e5b6c3d8528eac8f17423881f1"
Oct 31 02:19:03.952046 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:19:03.952029    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe2949ea3820d9e9e31cc129190a905dc7b124e5b6c3d8528eac8f17423881f1\": container with ID starting with fe2949ea3820d9e9e31cc129190a905dc7b124e5b6c3d8528eac8f17423881f1 not found: ID does not exist" containerID="fe2949ea3820d9e9e31cc129190a905dc7b124e5b6c3d8528eac8f17423881f1"
Oct 31 02:19:03.952096 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.952053    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe2949ea3820d9e9e31cc129190a905dc7b124e5b6c3d8528eac8f17423881f1"} err="failed to get container status \"fe2949ea3820d9e9e31cc129190a905dc7b124e5b6c3d8528eac8f17423881f1\": rpc error: code = NotFound desc = could not find container \"fe2949ea3820d9e9e31cc129190a905dc7b124e5b6c3d8528eac8f17423881f1\": container with ID starting with fe2949ea3820d9e9e31cc129190a905dc7b124e5b6c3d8528eac8f17423881f1 not found: ID does not exist"
Oct 31 02:19:03.952096 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.952068    5650 scope.go:122] "RemoveContainer" containerID="43b83ebd4624f35d9ba435a41f1d41d2f23b44cd8c5dce683dc9aeca5732f362"
Oct 31 02:19:03.958930 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.958907    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"]
Oct 31 02:19:03.962158 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.962142    5650 scope.go:122] "RemoveContainer" containerID="43b83ebd4624f35d9ba435a41f1d41d2f23b44cd8c5dce683dc9aeca5732f362"
Oct 31 02:19:03.962419 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:19:03.962402    5650 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43b83ebd4624f35d9ba435a41f1d41d2f23b44cd8c5dce683dc9aeca5732f362\": container with ID starting with 43b83ebd4624f35d9ba435a41f1d41d2f23b44cd8c5dce683dc9aeca5732f362 not found: ID does not exist" containerID="43b83ebd4624f35d9ba435a41f1d41d2f23b44cd8c5dce683dc9aeca5732f362"
Oct 31 02:19:03.962509 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.962426    5650 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b83ebd4624f35d9ba435a41f1d41d2f23b44cd8c5dce683dc9aeca5732f362"} err="failed to get container status \"43b83ebd4624f35d9ba435a41f1d41d2f23b44cd8c5dce683dc9aeca5732f362\": rpc error: code = NotFound desc = could not find container \"43b83ebd4624f35d9ba435a41f1d41d2f23b44cd8c5dce683dc9aeca5732f362\": container with ID starting with 43b83ebd4624f35d9ba435a41f1d41d2f23b44cd8c5dce683dc9aeca5732f362 not found: ID does not exist"
Oct 31 02:19:03.963316 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.963295    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-8d7k8"]
Oct 31 02:19:03.976824 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.976792    5650 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"]
Oct 31 02:19:03.981461 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:03.981423    5650 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-qmjjh"]
Oct 31 02:19:04.118462 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:04.118415    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4640ac90-9987-4201-a94e-56a3102f0f0e" path="/var/lib/kubelet/pods/4640ac90-9987-4201-a94e-56a3102f0f0e/volumes"
Oct 31 02:19:04.118909 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:04.118896    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af0ee471-3ebf-426f-88e8-689ddc123d8e" path="/var/lib/kubelet/pods/af0ee471-3ebf-426f-88e8-689ddc123d8e/volumes"
Oct 31 02:19:04.119334 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:04.119320    5650 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da800208-fbd5-460f-b1cc-a212a4445291" path="/var/lib/kubelet/pods/da800208-fbd5-460f-b1cc-a212a4445291/volumes"
Oct 31 02:19:04.798263 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:04.798227    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-544f7c9987-fwhzc_f0f46d02-6b50-40ec-b940-f767a9ce2b72/authorino/0.log"
Oct 31 02:19:15.757952 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:15.757916    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-75686f85d5-6f7td_4e714053-4cce-4192-a7a3-cc1ddb6c25fc/manager/0.log"
Oct 31 02:19:18.790024 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:18.789994    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-359bbaa1-z4nvh_73bed229-5ad1-415d-99a3-1abba0a327bc/revoke-keys/0.log"
Oct 31 02:19:19.337105 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:19.337072    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5b2209c2-rp8kf_1013a448-811b-4c57-ad4a-acb5d91553e4/revoke-keys/0.log"
Oct 31 02:19:19.885318 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:19.885293    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-96e50821-qlwlt_283f8e1a-8fae-4a8a-b75a-8fbdd5e563b5/revoke-keys/0.log"
Oct 31 02:19:22.371036 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:22.370991    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-544f7c9987-fwhzc_f0f46d02-6b50-40ec-b940-f767a9ce2b72/authorino/0.log"
Oct 31 02:19:25.834317 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:25.834243    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-nbn54_c3a0a419-29b7-4079-9511-96a84184790f/discovery/0.log"
Oct 31 02:19:26.236422 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:26.236387    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b44bd95d7-xxzfx_ac091ce7-c5c8-4c0a-936e-e3e7683c2841/kube-auth-proxy/0.log"
Oct 31 02:19:27.237229 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:27.237190    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-jpp7c_ced27998-939e-4734-9904-18c4dc0b1170/istio-proxy/0.log"
Oct 31 02:19:28.021745 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:28.021701    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-xljzj_a125e272-0ca1-47ca-9f22-e46a649b3f10/istio-proxy/0.log"
Oct 31 02:19:28.221737 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:28.221699    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-zznwh_b4ebd16b-3010-4a21-abd5-8b1f7e8c4905/istio-proxy/0.log"
Oct 31 02:19:28.418237 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:28.418157    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q_64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f/istio-proxy/0.log"
Oct 31 02:19:28.552195 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:19:28.552159    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:19:29.123304 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:29.123273    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-lq6ms_d5f6f417-2dd1-4f4e-ad96-92d1f512cac1/payload-processing/0.log"
Oct 31 02:19:30.515684 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:30.515653    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-g8s4b_1e31f257-42d1-4f16-9365-901371aa1356/main/0.log"
Oct 31 02:19:30.703220 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:30.703188    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-rmmmc_5321dae7-9deb-4010-99d4-2d9878bb62de/main/0.log"
Oct 31 02:19:31.239980 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:31.239950    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-rzqnw_966529d0-0bf9-4d10-a2f3-035558a1c051/main/0.log"
Oct 31 02:19:31.427106 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:31.427055    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-ldtbl_c7c22d93-6c25-479d-b877-15ed5e8ca249/main/0.log"
Oct 31 02:19:42.191968 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.191928    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7fz28/must-gather-hsjrl"]
Oct 31 02:19:42.192548 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192533    5650 cpu_manager.go:414] "containerMap: removing container" podUID="f14608d5-81ae-4ba3-8986-a8e602befe12" containerName="istio-proxy"
Oct 31 02:19:42.192609 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192551    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f14608d5-81ae-4ba3-8986-a8e602befe12" containerName="istio-proxy"
Oct 31 02:19:42.192609 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192568    5650 cpu_manager.go:414] "containerMap: removing container" podUID="da800208-fbd5-460f-b1cc-a212a4445291" containerName="istio-proxy"
Oct 31 02:19:42.192609 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192576    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da800208-fbd5-460f-b1cc-a212a4445291" containerName="istio-proxy"
Oct 31 02:19:42.192609 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192587    5650 cpu_manager.go:414] "containerMap: removing container" podUID="8969a12d-73b4-4d52-a144-918c72ded4bc" containerName="payload-pre-processing"
Oct 31 02:19:42.192609 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192597    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8969a12d-73b4-4d52-a144-918c72ded4bc" containerName="payload-pre-processing"
Oct 31 02:19:42.192772 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192614    5650 cpu_manager.go:414] "containerMap: removing container" podUID="73bed229-5ad1-415d-99a3-1abba0a327bc" containerName="revoke-keys"
Oct 31 02:19:42.192772 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192622    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73bed229-5ad1-415d-99a3-1abba0a327bc" containerName="revoke-keys"
Oct 31 02:19:42.192772 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192641    5650 cpu_manager.go:414] "containerMap: removing container" podUID="af0ee471-3ebf-426f-88e8-689ddc123d8e" containerName="istio-proxy"
Oct 31 02:19:42.192772 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192648    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af0ee471-3ebf-426f-88e8-689ddc123d8e" containerName="istio-proxy"
Oct 31 02:19:42.192772 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192663    5650 cpu_manager.go:414] "containerMap: removing container" podUID="dedb244a-f01c-463e-8201-c6d0ad8e3f3f" containerName="istio-proxy"
Oct 31 02:19:42.192772 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192671    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dedb244a-f01c-463e-8201-c6d0ad8e3f3f" containerName="istio-proxy"
Oct 31 02:19:42.192772 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192681    5650 cpu_manager.go:414] "containerMap: removing container" podUID="af0ee471-3ebf-426f-88e8-689ddc123d8e" containerName="istio-proxy"
Oct 31 02:19:42.192772 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192688    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af0ee471-3ebf-426f-88e8-689ddc123d8e" containerName="istio-proxy"
Oct 31 02:19:42.192772 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192700    5650 cpu_manager.go:414] "containerMap: removing container" podUID="4640ac90-9987-4201-a94e-56a3102f0f0e" containerName="payload-pre-processing"
Oct 31 02:19:42.192772 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192707    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4640ac90-9987-4201-a94e-56a3102f0f0e" containerName="payload-pre-processing"
Oct 31 02:19:42.192772 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192723    5650 cpu_manager.go:414] "containerMap: removing container" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main"
Oct 31 02:19:42.192772 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192730    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main"
Oct 31 02:19:42.192772 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192760    5650 cpu_manager.go:414] "containerMap: removing container" podUID="bb3603d0-f3c8-4bbc-96c7-366b59bffdce" containerName="istio-proxy"
Oct 31 02:19:42.192772 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192767    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3603d0-f3c8-4bbc-96c7-366b59bffdce" containerName="istio-proxy"
Oct 31 02:19:42.193225 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192780    5650 cpu_manager.go:414] "containerMap: removing container" podUID="f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" containerName="istio-proxy"
Oct 31 02:19:42.193225 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192787    5650 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" containerName="istio-proxy"
Oct 31 02:19:42.193225 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192868    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3603d0-f3c8-4bbc-96c7-366b59bffdce" containerName="istio-proxy"
Oct 31 02:19:42.193225 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192879    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="f14608d5-81ae-4ba3-8986-a8e602befe12" containerName="istio-proxy"
Oct 31 02:19:42.193225 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192891    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="da800208-fbd5-460f-b1cc-a212a4445291" containerName="istio-proxy"
Oct 31 02:19:42.193225 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192900    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="8969a12d-73b4-4d52-a144-918c72ded4bc" containerName="payload-pre-processing"
Oct 31 02:19:42.193225 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192911    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="73bed229-5ad1-415d-99a3-1abba0a327bc" containerName="revoke-keys"
Oct 31 02:19:42.193225 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192920    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="af0ee471-3ebf-426f-88e8-689ddc123d8e" containerName="istio-proxy"
Oct 31 02:19:42.193225 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192928    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="dedb244a-f01c-463e-8201-c6d0ad8e3f3f" containerName="istio-proxy"
Oct 31 02:19:42.193225 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192935    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="af0ee471-3ebf-426f-88e8-689ddc123d8e" containerName="istio-proxy"
Oct 31 02:19:42.193225 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192943    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="4640ac90-9987-4201-a94e-56a3102f0f0e" containerName="payload-pre-processing"
Oct 31 02:19:42.193225 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192951    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="a99af64c-978c-4d8c-9d5b-260fc08faf6d" containerName="main"
Oct 31 02:19:42.193225 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.192959    5650 memory_manager.go:365] "RemoveStaleState removing state" podUID="f93cf4d0-6dd7-40b5-ab25-a6ba6dc964db" containerName="istio-proxy"
Oct 31 02:19:42.198213 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.198180    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7fz28/must-gather-hsjrl"
Oct 31 02:19:42.202397 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.202367    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7fz28/must-gather-hsjrl"]
Oct 31 02:19:42.203075 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.203056    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7fz28\"/\"kube-root-ca.crt\""
Oct 31 02:19:42.203391 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.203366    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7fz28\"/\"openshift-service-ca.crt\""
Oct 31 02:19:42.205579 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.205511    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7fz28\"/\"default-dockercfg-8r2kr\""
Oct 31 02:19:42.271135 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.271094    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7a0ffda5-dbf5-4b7a-ac92-17c02e6eebef-must-gather-output\") pod \"must-gather-hsjrl\" (UID: \"7a0ffda5-dbf5-4b7a-ac92-17c02e6eebef\") " pod="openshift-must-gather-7fz28/must-gather-hsjrl"
Oct 31 02:19:42.271308 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.271209    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5qwc\" (UniqueName: \"kubernetes.io/projected/7a0ffda5-dbf5-4b7a-ac92-17c02e6eebef-kube-api-access-p5qwc\") pod \"must-gather-hsjrl\" (UID: \"7a0ffda5-dbf5-4b7a-ac92-17c02e6eebef\") " pod="openshift-must-gather-7fz28/must-gather-hsjrl"
Oct 31 02:19:42.371868 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.371824    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7a0ffda5-dbf5-4b7a-ac92-17c02e6eebef-must-gather-output\") pod \"must-gather-hsjrl\" (UID: \"7a0ffda5-dbf5-4b7a-ac92-17c02e6eebef\") " pod="openshift-must-gather-7fz28/must-gather-hsjrl"
Oct 31 02:19:42.372043 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.371887    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5qwc\" (UniqueName: \"kubernetes.io/projected/7a0ffda5-dbf5-4b7a-ac92-17c02e6eebef-kube-api-access-p5qwc\") pod \"must-gather-hsjrl\" (UID: \"7a0ffda5-dbf5-4b7a-ac92-17c02e6eebef\") " pod="openshift-must-gather-7fz28/must-gather-hsjrl"
Oct 31 02:19:42.372235 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.372211    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7a0ffda5-dbf5-4b7a-ac92-17c02e6eebef-must-gather-output\") pod \"must-gather-hsjrl\" (UID: \"7a0ffda5-dbf5-4b7a-ac92-17c02e6eebef\") " pod="openshift-must-gather-7fz28/must-gather-hsjrl"
Oct 31 02:19:42.385347 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.385319    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5qwc\" (UniqueName: \"kubernetes.io/projected/7a0ffda5-dbf5-4b7a-ac92-17c02e6eebef-kube-api-access-p5qwc\") pod \"must-gather-hsjrl\" (UID: \"7a0ffda5-dbf5-4b7a-ac92-17c02e6eebef\") " pod="openshift-must-gather-7fz28/must-gather-hsjrl"
Oct 31 02:19:42.516136 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.516093    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7fz28/must-gather-hsjrl"
Oct 31 02:19:42.802621 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.802590    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7fz28/must-gather-hsjrl"]
Oct 31 02:19:42.804111 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:19:42.804080    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a0ffda5_dbf5_4b7a_ac92_17c02e6eebef.slice/crio-4749ff1c548cb75cdb53c7c3ca402a8182c69a528c4514eae2ac5096fc927725 WatchSource:0}: Error finding container 4749ff1c548cb75cdb53c7c3ca402a8182c69a528c4514eae2ac5096fc927725: Status 404 returned error can't find the container with id 4749ff1c548cb75cdb53c7c3ca402a8182c69a528c4514eae2ac5096fc927725
Oct 31 02:19:42.806990 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:42.806969    5650 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 02:19:43.156711 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:43.156627    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7fz28/must-gather-hsjrl" event={"ID":"7a0ffda5-dbf5-4b7a-ac92-17c02e6eebef","Type":"ContainerStarted","Data":"4749ff1c548cb75cdb53c7c3ca402a8182c69a528c4514eae2ac5096fc927725"}
Oct 31 02:19:46.040696 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:46.040662    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5crfw_f8449422-f48d-4df2-8548-9de1c5339fd0/global-pull-secret-syncer/0.log"
Oct 31 02:19:46.452089 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:46.452058    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wnmfp_308d4c76-fad6-4dfd-8f55-5c5ac0d9b50f/konnectivity-agent/0.log"
Oct 31 02:19:46.511710 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:46.511667    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-98.us-east-2.compute.internal_7b4242bdffafd37f633de48cdce1cc09/haproxy/0.log"
Oct 31 02:19:48.789891 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:48.789852    5650 scope.go:122] "RemoveContainer" containerID="90eef6dc8c824bb334b0154a21eaa5728392d3b9d738c33426beb05137dd6cd8"
Oct 31 02:19:52.725101 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:52.725047    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-544f7c9987-fwhzc_f0f46d02-6b50-40ec-b940-f767a9ce2b72/authorino/0.log"
Oct 31 02:19:54.246537 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:54.246491    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7fz28/must-gather-hsjrl" event={"ID":"7a0ffda5-dbf5-4b7a-ac92-17c02e6eebef","Type":"ContainerStarted","Data":"e65bd402b656ec6c65bd291e71aa94ce059df10c104de705b9de28eb9229a56d"}
Oct 31 02:19:54.247010 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:54.246555    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7fz28/must-gather-hsjrl" event={"ID":"7a0ffda5-dbf5-4b7a-ac92-17c02e6eebef","Type":"ContainerStarted","Data":"76bb54b5d0896bf2760a32b6a8ada7b0b5a5569923c927a17e88ccb853c059b0"}
Oct 31 02:19:54.269258 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:54.269203    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-7fz28/must-gather-hsjrl" podStartSLOduration=1.588827497 podStartE2EDuration="12.269189329s" totalImagesPullingTime="10.680361832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:19:54.266038306 +0000 UTC m=+3550.269752599" watchObservedRunningTime="2026-10-31 02:19:54.269189329 +0000 UTC m=+3550.272903616"
Oct 31 02:19:54.887884 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:54.887853    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-75686f85d5-6f7td_4e714053-4cce-4192-a7a3-cc1ddb6c25fc/manager/0.log"
Oct 31 02:19:54.977319 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:54.977288    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/alertmanager/0.log"
Oct 31 02:19:55.003441 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:55.003405    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/config-reloader/0.log"
Oct 31 02:19:55.029857 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:55.029802    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/kube-rbac-proxy-web/0.log"
Oct 31 02:19:55.089732 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:55.089691    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/kube-rbac-proxy/0.log"
Oct 31 02:19:55.111662 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:55.111627    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/kube-rbac-proxy-metric/0.log"
Oct 31 02:19:55.140891 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:55.140803    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/prom-label-proxy/0.log"
Oct 31 02:19:55.169994 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:55.169936    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0d5d0ff7-7a5e-4a8e-916d-26e9c9572d80/init-config-reloader/0.log"
Oct 31 02:19:55.549762 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:55.549685    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cdkr4_de2a3af0-1220-4a82-a77a-772f99694285/node-exporter/0.log"
Oct 31 02:19:55.670131 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:55.670091    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cdkr4_de2a3af0-1220-4a82-a77a-772f99694285/kube-rbac-proxy/0.log"
Oct 31 02:19:55.735722 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:55.735688    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cdkr4_de2a3af0-1220-4a82-a77a-772f99694285/init-textfile/0.log"
Oct 31 02:19:55.838311 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:55.838231    5650 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kprhq/perf-node-gather-daemonset-zs7kb"]
Oct 31 02:19:55.844352 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:55.844324    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-zs7kb"
Oct 31 02:19:55.852991 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:55.852947    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kprhq/perf-node-gather-daemonset-zs7kb"]
Oct 31 02:19:55.855617 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:55.855580    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kprhq\"/\"kube-root-ca.crt\""
Oct 31 02:19:55.855617 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:55.855610    5650 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kprhq\"/\"default-dockercfg-2jdb2\""
Oct 31 02:19:55.856405 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:55.856384    5650 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kprhq\"/\"openshift-service-ca.crt\""
Oct 31 02:19:56.011098 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.011068    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8aa9f886-74c9-442f-89fc-beb5fc0bf260/prometheus/0.log"
Oct 31 02:19:56.022165 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.022135    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2mw4\" (UniqueName: \"kubernetes.io/projected/c289a1f8-4e79-4f09-8603-07237c44f1b0-kube-api-access-j2mw4\") pod \"perf-node-gather-daemonset-zs7kb\" (UID: \"c289a1f8-4e79-4f09-8603-07237c44f1b0\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-zs7kb"
Oct 31 02:19:56.022330 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.022172    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c289a1f8-4e79-4f09-8603-07237c44f1b0-sys\") pod \"perf-node-gather-daemonset-zs7kb\" (UID: \"c289a1f8-4e79-4f09-8603-07237c44f1b0\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-zs7kb"
Oct 31 02:19:56.022330 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.022244    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c289a1f8-4e79-4f09-8603-07237c44f1b0-lib-modules\") pod \"perf-node-gather-daemonset-zs7kb\" (UID: \"c289a1f8-4e79-4f09-8603-07237c44f1b0\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-zs7kb"
Oct 31 02:19:56.022427 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.022362    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c289a1f8-4e79-4f09-8603-07237c44f1b0-proc\") pod \"perf-node-gather-daemonset-zs7kb\" (UID: \"c289a1f8-4e79-4f09-8603-07237c44f1b0\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-zs7kb"
Oct 31 02:19:56.022427 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.022381    5650 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c289a1f8-4e79-4f09-8603-07237c44f1b0-podres\") pod \"perf-node-gather-daemonset-zs7kb\" (UID: \"c289a1f8-4e79-4f09-8603-07237c44f1b0\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-zs7kb"
Oct 31 02:19:56.037570 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.037535    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8aa9f886-74c9-442f-89fc-beb5fc0bf260/config-reloader/0.log"
Oct 31 02:19:56.067721 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.067687    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8aa9f886-74c9-442f-89fc-beb5fc0bf260/thanos-sidecar/0.log"
Oct 31 02:19:56.098177 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.098082    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8aa9f886-74c9-442f-89fc-beb5fc0bf260/kube-rbac-proxy-web/0.log"
Oct 31 02:19:56.123510 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.123470    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2mw4\" (UniqueName: \"kubernetes.io/projected/c289a1f8-4e79-4f09-8603-07237c44f1b0-kube-api-access-j2mw4\") pod \"perf-node-gather-daemonset-zs7kb\" (UID: \"c289a1f8-4e79-4f09-8603-07237c44f1b0\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-zs7kb"
Oct 31 02:19:56.123740 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.123523    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c289a1f8-4e79-4f09-8603-07237c44f1b0-sys\") pod \"perf-node-gather-daemonset-zs7kb\" (UID: \"c289a1f8-4e79-4f09-8603-07237c44f1b0\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-zs7kb"
Oct 31 02:19:56.123740 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.123582    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c289a1f8-4e79-4f09-8603-07237c44f1b0-lib-modules\") pod \"perf-node-gather-daemonset-zs7kb\" (UID: \"c289a1f8-4e79-4f09-8603-07237c44f1b0\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-zs7kb"
Oct 31 02:19:56.123740 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.123644    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c289a1f8-4e79-4f09-8603-07237c44f1b0-proc\") pod \"perf-node-gather-daemonset-zs7kb\" (UID: \"c289a1f8-4e79-4f09-8603-07237c44f1b0\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-zs7kb"
Oct 31 02:19:56.123740 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.123668    5650 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c289a1f8-4e79-4f09-8603-07237c44f1b0-podres\") pod \"perf-node-gather-daemonset-zs7kb\" (UID: \"c289a1f8-4e79-4f09-8603-07237c44f1b0\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-zs7kb"
Oct 31 02:19:56.124002 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.123761    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c289a1f8-4e79-4f09-8603-07237c44f1b0-sys\") pod \"perf-node-gather-daemonset-zs7kb\" (UID: \"c289a1f8-4e79-4f09-8603-07237c44f1b0\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-zs7kb"
Oct 31 02:19:56.124002 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.123798    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c289a1f8-4e79-4f09-8603-07237c44f1b0-podres\") pod \"perf-node-gather-daemonset-zs7kb\" (UID: \"c289a1f8-4e79-4f09-8603-07237c44f1b0\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-zs7kb"
Oct 31 02:19:56.124002 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.123812    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c289a1f8-4e79-4f09-8603-07237c44f1b0-lib-modules\") pod \"perf-node-gather-daemonset-zs7kb\" (UID: \"c289a1f8-4e79-4f09-8603-07237c44f1b0\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-zs7kb"
Oct 31 02:19:56.124002 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.123822    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c289a1f8-4e79-4f09-8603-07237c44f1b0-proc\") pod \"perf-node-gather-daemonset-zs7kb\" (UID: \"c289a1f8-4e79-4f09-8603-07237c44f1b0\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-zs7kb"
Oct 31 02:19:56.124621 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.124591    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8aa9f886-74c9-442f-89fc-beb5fc0bf260/kube-rbac-proxy/0.log"
Oct 31 02:19:56.137222 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.137188    5650 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2mw4\" (UniqueName: \"kubernetes.io/projected/c289a1f8-4e79-4f09-8603-07237c44f1b0-kube-api-access-j2mw4\") pod \"perf-node-gather-daemonset-zs7kb\" (UID: \"c289a1f8-4e79-4f09-8603-07237c44f1b0\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-zs7kb"
Oct 31 02:19:56.152133 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.152104    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8aa9f886-74c9-442f-89fc-beb5fc0bf260/kube-rbac-proxy-thanos/0.log"
Oct 31 02:19:56.160305 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.160281    5650 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-zs7kb"
Oct 31 02:19:56.178798 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.178762    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8aa9f886-74c9-442f-89fc-beb5fc0bf260/init-config-reloader/0.log"
Oct 31 02:19:56.222347 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.222316    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-97j2h_c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243/prometheus-operator/0.log"
Oct 31 02:19:56.260157 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.260125    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-97j2h_c2b2a6ce-eb1d-4c15-a5cc-3a09370eb243/kube-rbac-proxy/0.log"
Oct 31 02:19:56.438929 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.438836    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86d478dd89-l66f7_76b7d8bd-5205-4243-a45f-c208a65bbc30/thanos-query/0.log"
Oct 31 02:19:56.451795 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.451760    5650 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kprhq/perf-node-gather-daemonset-zs7kb"]
Oct 31 02:19:56.457372 ip-10-0-133-98 kubenswrapper[5650]: W1031 02:19:56.457326    5650 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc289a1f8_4e79_4f09_8603_07237c44f1b0.slice/crio-dce504443d51fb33812e71325045727b8975678e81dade69d2de53f2d5fb4e0e WatchSource:0}: Error finding container dce504443d51fb33812e71325045727b8975678e81dade69d2de53f2d5fb4e0e: Status 404 returned error can't find the container with id dce504443d51fb33812e71325045727b8975678e81dade69d2de53f2d5fb4e0e
Oct 31 02:19:56.481053 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.480998    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86d478dd89-l66f7_76b7d8bd-5205-4243-a45f-c208a65bbc30/kube-rbac-proxy-web/0.log"
Oct 31 02:19:56.508648 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.508614    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86d478dd89-l66f7_76b7d8bd-5205-4243-a45f-c208a65bbc30/kube-rbac-proxy/0.log"
Oct 31 02:19:56.535735 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.535706    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86d478dd89-l66f7_76b7d8bd-5205-4243-a45f-c208a65bbc30/prom-label-proxy/0.log"
Oct 31 02:19:56.564198 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.564161    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86d478dd89-l66f7_76b7d8bd-5205-4243-a45f-c208a65bbc30/kube-rbac-proxy-rules/0.log"
Oct 31 02:19:56.597724 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.597504    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86d478dd89-l66f7_76b7d8bd-5205-4243-a45f-c208a65bbc30/kube-rbac-proxy-metrics/0.log"
Oct 31 02:19:56.613552 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:56.613489    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-75686f85d5-6f7td_4e714053-4cce-4192-a7a3-cc1ddb6c25fc/manager/0.log"
Oct 31 02:19:57.272462 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:57.272413    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-zs7kb" event={"ID":"c289a1f8-4e79-4f09-8603-07237c44f1b0","Type":"ContainerStarted","Data":"d5f45a2c6598480362c17def263e8b1b3d6092d5129f850e8305aab6c1eac7da"}
Oct 31 02:19:57.272677 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:57.272479    5650 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-zs7kb" event={"ID":"c289a1f8-4e79-4f09-8603-07237c44f1b0","Type":"ContainerStarted","Data":"dce504443d51fb33812e71325045727b8975678e81dade69d2de53f2d5fb4e0e"}
Oct 31 02:19:57.272677 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:57.272628    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-zs7kb"
Oct 31 02:19:57.293791 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:57.293736    5650 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-zs7kb" podStartSLOduration=2.293721182 podStartE2EDuration="2.293721182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:19:57.292693968 +0000 UTC m=+3553.296408262" watchObservedRunningTime="2026-10-31 02:19:57.293721182 +0000 UTC m=+3553.297435472"
Oct 31 02:19:57.952343 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:57.952298    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-75686f85d5-6f7td_4e714053-4cce-4192-a7a3-cc1ddb6c25fc/manager/0.log"
Oct 31 02:19:58.561274 ip-10-0-133-98 kubenswrapper[5650]: E1031 02:19:58.561229    5650 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8n2r2" containerName="extractor"
Oct 31 02:19:59.800515 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:19:59.800476    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-75686f85d5-6f7td_4e714053-4cce-4192-a7a3-cc1ddb6c25fc/manager/0.log"
Oct 31 02:20:00.879790 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:20:00.879751    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-75686f85d5-6f7td_4e714053-4cce-4192-a7a3-cc1ddb6c25fc/manager/0.log"
Oct 31 02:20:01.717984 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:20:01.717946    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kmxk8_aa56c5f6-4784-4b24-9daa-d17f7b296d8b/dns/0.log"
Oct 31 02:20:01.744818 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:20:01.744785    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kmxk8_aa56c5f6-4784-4b24-9daa-d17f7b296d8b/kube-rbac-proxy/0.log"
Oct 31 02:20:01.917913 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:20:01.917878    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xghmx_86947040-78ac-43d5-b932-9acd63cf43e7/dns-node-resolver/0.log"
Oct 31 02:20:02.673712 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:20:02.673675    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tzz5q_52e25fa9-6c04-453a-a0b2-cca32a428a5a/node-ca/0.log"
Oct 31 02:20:03.189041 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:20:03.188999    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-75686f85d5-6f7td_4e714053-4cce-4192-a7a3-cc1ddb6c25fc/manager/0.log"
Oct 31 02:20:03.290914 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:20:03.290879    5650 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-zs7kb"
Oct 31 02:20:12.775213 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:20:12.775169    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-nbn54_c3a0a419-29b7-4079-9511-96a84184790f/discovery/0.log"
Oct 31 02:20:12.811420 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:20:12.811386    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b44bd95d7-xxzfx_ac091ce7-c5c8-4c0a-936e-e3e7683c2841/kube-auth-proxy/0.log"
Oct 31 02:20:12.944365 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:20:12.944327    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-jpp7c_ced27998-939e-4734-9904-18c4dc0b1170/istio-proxy/0.log"
Oct 31 02:20:13.051819 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:20:13.051721    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-xljzj_a125e272-0ca1-47ca-9f22-e46a649b3f10/istio-proxy/0.log"
Oct 31 02:20:13.086653 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:20:13.086610    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-zznwh_b4ebd16b-3010-4a21-abd5-8b1f7e8c4905/istio-proxy/0.log"
Oct 31 02:20:13.106641 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:20:13.106574    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64487b45b7-zxz9q_64c1826b-d7f2-4ef7-8d5b-4c1b4a7d5c3f/istio-proxy/0.log"
Oct 31 02:20:13.152790 ip-10-0-133-98 kubenswrapper[5650]: I1031 02:20:13.152729    5650 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-lq6ms_d5f6f417-2dd1-4f4e-ad96-92d1f512cac1/payload-processing/0.log"